Extract data from the "Share and Multiply" dataset for use with MGen.
|
2 months ago | |
---|---|---|
data | 2 months ago | |
hmm | 2 months ago | |
src | 2 months ago | |
Cargo.toml | 10 months ago | |
README.md | 2 months ago |
This repo contains tools to extract empirical distributions from the "Share and Multiply" (SaM) dataset of WhatsApp chat metadata.
More thorough documentation is coming soon, but the gist is:
json_files.zip
file they provide, and extract it somewhere.extract
tool to pare and serialize the SaM data.
(Using chat*.json
in any of the following commands means using all available chats; you can use a subset for faster processing, so long as you're consistent.)
cargo run --bin extract stats/ json_files/chat*.json
hmm
to label messages as "active" or "idle".
pip install -r requirements.txt
./parallel_run.sh ../stats/ stats2/
process
tool to generate all empirical distributions other than message sizes.
cargo run --bin process dists/ hmm/stats2/ json_files/chat*.json
message-lens
tool to generate distributions for message sizes.
This takes an optional argument for file sizes (must be first if provided, sorry for the jank).
If you have a source for file sizes, you can provide it here.
If you don't want to simulate sending files, you can omit it.
If you don't have a source, you can use the one we provide based on public WhatsApp groups in 2023.
cargo run --bin message-lens -- -s data/file_sizes.dat dists/ json_files/chat*.json
At this point, dists/
will contain distributions ready for use in MGen, organized by the user being simulated.
We also provide some data that may be useful.
As mentioned above, data/file_sizes.dat
contains our own findings for distribution of file sizes in public WhatsApp groups as monitored in 2023.
The data/dyadic_count.dat
and data/group_count.dat
files give the relative fraction of the number of dyadic conversations (i.e., one-on-one conversations) and group conversations users were in, respectively, as sourced from "Analysis of Group-Based Communication in WhatsApp".
The data/group_sizes.dat
file lists the absolute frequencies of each possible group size in WhatsApp, from 0 to 255, from the SaM dataset.
The data/group_sizes.no_individual.dat
file is the same, but with groups of size 0, 1, and 2 all set to a count of 0, as they are better modeled by the dyadic data (this file also includes a second row enumerating 0 to 255 for easier use with MGen-style parsing of the distribution).