Science Memes

11021 readers

3049 users here now

Welcome to c/science_memes @ Mander.xyz!

A place for majestic STEMLORD peacocking, as well as memes about the realities of working in a lab.

Rules

Don't throw mud. Behave like an intellectual and remember the human.
Keep it rooted (on topic).
No spam.
Infographics welcome, get schooled.

This is a science community. We use the Dawkins definition of meme.

Research Committee

[email protected]

Other Mander Communities

Science and Research

Biology and Life Sciences

Physical Sciences

Humanities and Social Sciences

Practical and Applied Sciences

Memes

Miscellaneous

founded 2 years ago

MODERATORS

[email protected]

341

Pandas (mander.xyz)

submitted 3 months ago by [email protected] to c/[email protected]

36 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 8 points 3 months ago (1 children)

Pandas supports lazy loading and can read files in chunks. Hell, even regular ole Python doesn't need to read the whole file at once with csv

[–] [email protected] 3 points 3 months ago* (last edited 3 months ago)

I didn't know about lazy loading, that's cool!

Then I guess that the meme doesn't apply anymore. Though I will state that (from my anedoctal experience) people that can use Panda's most advanced features* are also comfortable with other data processing frameworks (usually more suitable to large datasets**)

*Anything beyond the standard groupby - apply can be considered advanced, from the placrs I've been

**I feel the urge to note that 60Mb isn' lt a large dataset by any means, but I believe that's beyond the point