7
submitted 21 hours ago* (last edited 21 hours ago) by [email protected] to c/[email protected]

I've noticed some files I opened in a text editor have all kinds of crazy unrenderable chars

top 21 comments
sorted by: hot top controversial new old
[-] [email protected] 16 points 21 hours ago
[-] [email protected] 5 points 21 hours ago* (last edited 21 hours ago)
[-] [email protected] 3 points 16 hours ago
[-] [email protected] 8 points 21 hours ago

can it? Sure, most any arrangement of bits can be converted into some kind of Unicode text. Can it be converted to something meaningful or readable? No, some formats are plain text (.txt, .ini, .json, .html for some random examples) that are meant to be read by humans, and others are binary formats that are only meaningful when decoded by a computer into specific data structures inside a piece of software.

[-] [email protected] 4 points 21 hours ago* (last edited 21 hours ago)
[-] [email protected] 1 points 21 hours ago* (last edited 21 hours ago)

Can you comment on the specific makeup of a "rendered" audio file in plaintext, how is the computer representing every little noise bit of sound at any given point, the polyphony etc?

What are the conventions of such representation? How can a spectrogram tell pitches are where they are, how is the computer representing that?

Is it the same to view plaintext as analysing it with a hex-viewer?

[-] [email protected] 6 points 20 hours ago

There's two things at play here.

MP3 (or WAV, OGG, FLAC etc.) provide a way to encode polyphony and stereo and such into a sequence of bytes.

And then separately, there's Unicode (or ASCII) for encoding letters into bytes. These are just big tables which say e.g.:

  • 01000001 = uppercase 'A'
  • 01000010 = uppercase 'B'
  • 01100001 = lowercase 'A'

So, what your text editor does, is that it looks at the sequence of bytes that MP3 encoded and then it just looks into its table and somewhat erronously interprets it as individual letters.

[-] [email protected] 4 points 20 hours ago

I think you are conflating a few different concepts here.

Can you comment on the specific makeup of a “rendered” audio file in plaintext, how is the computer representing every little noise bit of sound at any given point, the polyphony etc?
What are the conventions of such representation? How can a spectrogram tell pitches are where they are, how is the computer representing that?

This is a completely separate concern from how data can be represented as text, and will vary by audio format. The "simplest", PCM encoded audio like in a .wav file, doesn't really concern itself at all with polyphony and is just a quantised representation of the audio wave amplitude at any given instant in time. It samples that tens of thousands of times per second. Whether it's a single pure tone or a full symphony the density of what's stored is the same. Just an air-pressure-over-time graph, essentially.

Is it the same to view plaintext as analysing it with a hex-viewer?

"Plaintext" doesn't really have a fixed definition in this context. It can be the same as looking at it in a hex viewer, if your "plaintext" representation is hexadecimal encoding. Binary data, like in audio files, isn't plaintext, and opening it directly in a text editor is not expected to give you a useful result, or even a consistent result. Different editors might show you different "text" depending on what encoding they fall back on, or how they represent unprintable characters.

There are several methods of representing binary data as text, such as hexadecimal, base64, or uuencode, but none of these representations if saved as-is are the original file, strictly speaking.

[-] [email protected] 3 points 21 hours ago

Most binary-to-text encodings don’t attempt to make the text human-readable—they’re just intended to transmit the data over a text-only medium to a recipient who will decode it back to the original binary format.

[-] [email protected] 1 points 20 hours ago* (last edited 20 hours ago)

I do understand I'm not able to read it myself, I'm more curious about the architecture of how that data is represented and stored and conceptually how such representation is practically organized/reified...

[-] [email protected] 3 points 20 hours ago* (last edited 6 hours ago)

The original binary format is split into six-bit chunks (e.g., 100101), which in decimal format correspond to the integers from 0 to 63. These are just mapped to letters in order:

  1. 000000 = A,
  2. 000001 = B,
  3. 000010 = C,
  4. 000011 = D,

etc.—it goes through the capital letters first, then lower-case letters, then digits, then “+” and “/”. It’s so simple you could do it by hand from the above description, if you were looking at the data in binary format.

[-] [email protected] 4 points 21 hours ago

technically, yes. all unprintable binary can be resolved to 64 printable characters. but that resulting string may not be english or any human language.

[-] [email protected] 1 points 21 hours ago* (last edited 21 hours ago)

But its still contains the actual data in a faithfully reproducible/useable way?

[-] [email protected] 6 points 21 hours ago

Yes. Decoding a base64 encoded string will give you back the exact original data.

Importantly though, this isn't what you're seeing when you open files in a text editor as you describe in your original post, and if you copied the text of those files and saved a new copy it's very likely that it would not reproduce correctly.

[-] [email protected] 4 points 21 hours ago

yes, this method doesn't lose any bits. one of its primary use before was email which was strictly text only.

[-] [email protected] 3 points 21 hours ago

Are those binary files by any chance?

[-] [email protected] 2 points 21 hours ago* (last edited 21 hours ago)

I just mean like any file (pdf, jpeg, mp4, mp3, exe—

mp4/mp3 most famously for me

I find it so damn cool and incredible I can record something/anything right now and open the audio in a text file and its all right there—albeit in an incomprehensible format but there altogether.

Its like a thinking rock etching sound into stone

[-] [email protected] 8 points 21 hours ago* (last edited 20 hours ago)

If you're on Linux, you can convert that to something more human readable by piping it to base64. It works with any file, but I'll use an image here:

cat image.webp | base64

Which yields:

UklGRroEAABXRUJQVlA4WAoAAAAgAAAAYwAAQgAASUNDUKACAAAAAAKgbGNtcwRAAABtbnRyUkdC
IFhZWiAH6AAIABoADgAJACBhY3NwQVBQTAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA9tYAAQAA
AADTLWxjbXMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA1k
ZXNjAAABIAAAAEBjcHJ0AAABYAAAADZ3dHB0AAABmAAAABRjaGFkAAABrAAAACxyWFlaAAAB2AAA
ABRiWFlaAAAB7AAAABRnWFlaAAACAAAAABRyVFJDAAACFAAAACBnVFJDAAACFAAAACBiVFJDAAAC
FAAAACBjaHJtAAACNAAAACRkbW5kAAACWAAAACRkbWRkAAACfAAAACRtbHVjAAAAAAAAAAEAAAAM
ZW5VUwAAACQAAAAcAEcASQBNAFAAIABiAHUAaQBsAHQALQBpAG4AIABzAFIARwBCbWx1YwAAAAAA
AAABAAAADGVuVVMAAAAaAAAAHABQAHUAYgBsAGkAYwAgAEQAbwBtAGEAaQBuAABYWVogAAAAAAAA
9tYAAQAAAADTLXNmMzIAAAAAAAEMQgAABd7///MlAAAHkwAA/ZD///uh///9ogAAA9wAAMBuWFla
IAAAAAAAAG+gAAA49QAAA5BYWVogAAAAAAAAJJ8AAA+EAAC2xFhZWiAAAAAAAABilwAAt4cAABjZ
cGFyYQAAAAAAAwAAAAJmZgAA8qcAAA1ZAAAT0AAACltjaHJtAAAAAAADAAAAAKPXAABUfAAATM0A
AJmaAAAmZwAAD1xtbHVjAAAAAAAAAAEAAAAMZW5VUwAAAAgAAAAcAEcASQBNAFBtbHVjAAAAAAAA
AAEAAAAMZW5VUwAAAAgAAAAcAHMAUgBHAEJWUDgg9AEAALAQAJ0BKmQAQwA+8WSmTqmlKCYvmWqp
MB4JZQDLnNaF2NMD2L3xQGb5nmLiGhGWxQuD8kwUSXF0u2UTgX0YrR3MY2SsRCNEQ8hZ6WkCUTih
LdmsElHZVzoMwO/fj4X/ZSNT2R9qgxwqgEed891j4KCNRLK/tUbG3hZ3Mw2kixguSFIEcAgBtv8w
eAu0PwAA/upMzBqq+dcN8viO7FpqpV6GvPcRILm+HsOQblnpHx03lASjGlSyGbkKUD3xA5KOqgq/
VEUJ4qF9VoAYFbFhQRAgkvmREk5umMj8sr9Np95+n/oP2Aq2VW5xU4F1xpD8Vd4Dp7Phwm9w/Dnf
94djRROFRYPZeg/1Q/qiROFRVRu2nBcgndbhc0x0h+kgvT/naeJOEqwNjYPlIiw/DGuxav7+x09R
mf2mJto3ineDqfyMWUN83PmKqzGHkYGhZrTU478qjlQucDzWkwobnUmzhE6I+mDYkfiUVPcHyXbf
xXRStyPiPZAkJZrE9OrjFNUeljRQdVTQqeBsy+O9VwDLU5GcKhBQHa4cj+/DGqUhi74WH0EuHsb3
EgZVNc1FbRm5QFOpjDSprGIRYxe6sFFDrDOg4DhWZRnOa7s68pGaDDpbqrORxzPHXPbs55/1HTas
DDGzKFmTG4hJ2GUZKqjPcQ+MAAAA

Copy that into a text file and pass it to base64 with the decode flag, and you'll get the original binary:

cat data.txt | base64 -d > data.bin

Inspect it to see what kind of file it is:

file data.bin -> data.bin: RIFF (little-endian) data, Web/P image

Rename it so you can just double-click it to open it:

mv data.bin data.webp

Enjoy the surprise.

You can also print files like that, scan them using OCR, and then restore them. A very inefficient way to do backups, but it works.

[-] [email protected] 3 points 20 hours ago

How is it representing it tho? Like does it have woven in there an array of hexcode colors for every microscopic pixel that makea up the picture.

Are images and audio files just arrays of frames which are arrays of pixels and sound units?

[-] [email protected] 4 points 20 hours ago* (last edited 20 hours ago)

It just converts the raw binary data into character encoding, so it doesn't matter what the source is (image, video, database file, etc). The source binary data is taken 6 bits at a time, then this group of 6 bits is mapped to one of 64 unique characters.

The decoding process is just the reverse of that: mapping the data back to binary form.

https://en.wikipedia.org/wiki/Base64

[-] [email protected] 1 points 18 hours ago

You can use a hex editor to view those files and even change them in some cases.

Something like this https://github.com/WerWolv/ImHex

this post was submitted on 20 Sep 2024
7 points (100.0% liked)

Ask Lemmy

26285 readers
1269 users here now

A Fediverse community for open-ended, thought provoking questions


Rules: (interactive)


1) Be nice and; have funDoxxing, trolling, sealioning, racism, and toxicity are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them


2) All posts must end with a '?'This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?


3) No spamPlease do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.


4) NSFW is okay, within reasonJust remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either [email protected] or [email protected]. NSFW comments should be restricted to posts tagged [NSFW].


5) This is not a support community.
It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email [email protected]. For other questions check our partnered communities list, or use the search function.


Reminder: The terms of service apply here too.

Partnered Communities:

Tech Support

No Stupid Questions

You Should Know

Reddit

Jokes

Ask Ouija


Logo design credit goes to: tubbadu


founded 1 year ago
MODERATORS