Data Engineering

185 readers

1 users here now

Discussion on Data Engineering topics. Data pipelines, tools and technologies, databases and DBMS, best practices:

Rules:

Limited to data engineering, no general CS/programming posts.
No technical questions. Example: how to fix this bug in my code.
No marketing
No resumes, jobs
No PII

founded 1 year ago

MODERATORS

[email protected]

How do I convince my data engineer to not modify data before including it in our db? (lemm.ee)

submitted 1 year ago by [email protected] to c/[email protected]

8 comments fedilink hide all child comments

Our data engineer insists in lowercasing everything and removing some other formatting like new lines on free text fields.

They say it's "better for elastic search".

To me that makes no sense and loses information that can't be added back. But I couldn't really convince them otherwise. So far no real problem has come out of it but it makes for a worse experience for the user. Like company names that are acronyms show up as all lowercase. (ibm, llc, etc.) or free text fields that we miss when the user wrote in caps or added paragraphs.

What are your thoughts on this?

Disclaimer, I'm not a data engineer. Just a PM from a data related product.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 1 points 1 year ago

It is fine if your database has _A tables (also called journal or audit tables) as the previous values would be stored in the _A table entries in case you ever desired to get that data back.

But if your database is missing such good practices, tell them to just use lower() or upper() and leave your data alone