The app seems to contact external IP addresses. Could you tell me why this is?

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
[X] I've searched the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
Hello,
I wonder if you could provide insight into why newer versions of yt-dlp contact external IP addresses? If you compare and contrast the "Relations" tab on github you will notice the difference.
Version 2022.11.11:
https://www.virustotal.com/gui/file/3e73e80d5b29be0a6bd47cb8636d281ce8a53ba11533f3441ea7d12433165b41/relations
Version 2022.02.04:
https://www.virustotal.com/gui/file/030ef7cbf3141d02a499ed9a4a4181df591efb1f516e7e4a2ef3fa9d83d5e67a/relations
If I was to compile the app myself can I remove this functionality from it?
Kind regards.
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

I cannot reproduce the behavior on local system. You'd have to contact virustotal to know what exactly is happening.
All the source code and build workflow is public - so you may inspect it as well if desired

Happy New Year, everybody!
As I don't host any media server in my internal network, yt-dlp has to make a connection to external servers, I don't have control over (e.g Youtube). Maybe some sites have redirects (depending on user agent string) to IP addresses considered suspicious by Virustotal.

I assume they would be testing by just running yt-dlp, not yt-dlp URL

Nowadays safety and security are an illusion, given by third party "certifications"...
https://processchecker.com/file/yt-dlp.exe.html

That would just be background traffic on the Windows VM that VirusTotal initialized for the scan.
Possibly SmartScreen or some other dumb feature that causes the VM to contact a bunch of Microsoft IPs when the application is run. No, VirusTotal don't filter that stuff out.

I'm not sure what you're talking about here, because when I switch to the "Behavior" tab, I see "Network comms: NOT FOUND", and the "Relations" tab that OP is linking to is only listing "Execution Parents", which seem to be other applications using yt-dlp under the hood.

I'm not sure what you're talking about here, because when I switch to the "Behavior" tab, I see "Network comms: NOT FOUND", and the "Relations" tab that OP is linking to is only listing "Execution Parents", which seem to be other applications using yt-dlp under the hood.

I think you were talking to OP, but VirusTotal's "Behavior" tab specifies:

Network comms
1 DNS 12 IP

As stated though, this issue is almost certainly nothing of concern. VirusTotal sandboxes don't try to differentiate between network traffic the application specifically sent and network traffic the VM sent at the direction of another process, possibly as a result of malicious behavior from the submitted sample. In this case it recorded some innocuous Windows feature as it's known to do.

ah, I looked at the second link thinking it was the latest version.
all of the IP addresses listed belong to either AS8075 (Microsoft Corporation) or AS20940 (Akamai International B.V.).
the first one is obvious. the second might not be, but Microsoft was cooperating with Akamai back in 1999, still partners with Akamai for CDN services, and especially - domains pointing to Akamai are listed by Microsoft as Windows services, including: "Enables connections to Windows Update", "Windows Defender", "Microsoft Store".

I wish you a belated happy new year lol. I did not get a response from virustotal. However I noticed the latest version also contacts domains and a comment was left explaining why the app could be doing this. Perhaps it could be compiled differently?

This host is linked to Microsoft, and/or Microsoft application tracking. Because for instance, if you build a clean EXE in Visual Studio 2022 (my test case was: building "Windows Desktop Application" template without further modifications, in Release x64 mode) you'll end up with a main application binary that contacts this host (through its IP address <IP_ADDRESS> on TCP 80 while forming the DNS resolutions fp2e7a.wpc.2be4.phicdn.net and fp2e7a.wpc.phicdn.net) according to VirusTotal, therefore linking the EXE against it.

Of course, the same can happen with malware built using Visual Studio, or another environment that integrates something of which this is part. #EdgeCast (registrar of contacted host's IP <IP_ADDRESS>) is clearly connected to #Microsoft and #VisualStudio produced applications in this context.
All considered, the IP alone (or its tree of correlations) isn't an indicator of malicious or not malicious. It's extremely generic to be placed on tons of application EXE files.
#fp2e7a.wpc.2be4.phicdn.net
#fp2e7a.wpc.phicdn.net
#phicdn part of #EdgeCast and tied to #Microsoft Visual Studio produced applications/EXE binaries, the nature of traffic could be telemetry directed by Microsoft
For the record, my test case is file SHA256 58fc9286c3aac58103b44af0bfaf798c7f404da8a30eb326d24fa5835ff89839 and you can as well see what i describe in VT behavior tab and from following correlations
// Update: in web page form, this host (at http://fp2e7a.wpc.2be4.phicdn.net/) calls itself a "CRL/CACERT Repository" so it can be a side station of Visual Studio-produced applications telemetry flow/stack.
https://www.virustotal.com/gui/domain/fp2e7a.wpc.2be4.phicdn.net/community
It also has this listed against your app:
Joe Sandbox Analysis:
Verdict: SUS
Score: 29/100
HTML Report: https://www.joesandbox.com/analysis/832895/0/html
PDF Report: https://www.joesandbox.com/analysis/832895/0/pdf
Executive Report: https://www.joesandbox.com/analysis/832895/0/executive
Incident Report: https://www.joesandbox.com/analysis/832895/0/irxml
IOCs: https://www.joesandbox.com/analysis/832895?idtype=analysisid
Thanks.

Can you reproduce on a local system? I cannot


Incompatibility with dburles:collection-helpers

Hi, trying to use helpers from dburles:collection-helpers and got this error:
Uncaught Error: [Can't apply helpers to 'meals' a transform function already exists!]

Any way to make them compatible?

This issue is more dburles:collection-helpers than tap-i18n-db


refactor(test.js): Refine test runner and npm scripts

test.js script should be easier to understand
test:once script is removed because it's not needed and there are no mocha tests

I hope I was right that test:once is no longer needed, please correct me if I'm wrong, I'll revert the change and add a documentation on what's the purpose of it.

Rebased, reformatted.


[WIP] client-side proxy

I discovered securityContext: runAsUid which lets me get rid of permission stuff.


test(Case Management): use pagination with get_cases operation

PR summary
This PR is a simple change to the Case Management integration test to use pagination with the get_cases() operation.
PR Checklist
Please make sure that your PR fulfills the following requirements:

[ ] The commit message follows the Angular Commit Message Guidelines.
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] New tests
[ ] Build/CI related changes
[ ] Documentation content changes
[ ] Other (please describe)

What is the current behavior?

What is the new behavior?

Does this PR introduce a breaking change?

[ ] Yes
[ ] No

Other information

:tada: This PR is included in version 0.17.9 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Cassandra preserve_host error after restarting Kong

After restarting the official docker container Kong 0.4.2 seems to want to update Casandra. I haven't updated the version.
[INFO] Connecting to the database...
[ERR] Cassandra error: Cassandra returned error (Invalid): Invalid column name preserve_host because it conflicts with an existing column

My migration table looks like this :
 id         | migrations
------------+------------------------------------------------------------------------------------------------------------------------------
 migrations | ['2015-01-12-175310_init_schema', '2015-01-12-175310_init_schema', '2015-04-24-154530_plugins', '2015-04-24-154530_plugins']

There is something very wrong with your schema_migrations table. If I believe the error, its content should be:
 id         | migrations
------------+-------------------------------------------------------------------------------------------------------------------------------------------------
 migrations | ['2015-01-12-175310_init_schema', '2015-04-24-154530_plugins', '2015-05-22-235608_0.3.0', '2015-06-09-170921_0.4.0', '2015-08-10-813213_0.4.2']

I don't know what went wrong during your previous installation/migrations to cause this. A way to fix it would be to execute the following queries with cqlsh:
DELETE FROM schema_migrations WHERE id = 'migrations';
INSERT INTO schema_migrations (id, migrations) VALUES('migrations', ['2015-01-12-175310_init_schema', '2015-04-24-154530_plugins', '2015-05-22-235608_0.3.0', '2015-06-09-170921_0.4.0', '2015-08-10-813213_0.4.2']);

Note that when 0.5.0 will be released, the schema changes will require that you run a python script to migrate your database, that will definitely be something to watch closely.

Thanks. The interesting thing is that I did not upgrade. The Kong instances were shutdown abruptly due to a scheduler glitch but were restarted on the same version.
I think I have an idea where the error might have originated from. We are using the Marathon  scheduler from Mesosphere with two instances. They get started at the same time on multiple machines. If there is a migration to be done both instances are competing to update the DB as they are doing this on start-up, this might explain the double entries in the migration table. I'm under the impression that the script is not thread safe.
What do you think? A lock flag in the DB itself could prevent this behavior.

Yes that is an issue we had with our AWS AMI and cloudformation distributions too. I always wanted the migration step to be manual $ kong init && kong start but having the migrations bundled in kong start is what was decided...
As for a lock in the DB, well I think it would only move the problem, what if the two instances create the lock at the same time?
Another solution I thought for this is to document this edgecase, explain to run the $ kong migrations up manually and then run kong start --no-migrations, and this flag prevents the command from running the migrations. I still think the first solution is cleaner and more common though...

@thibaultCha any updates?

Closing this as it's an old version of Kong and an old ticket. If the behaviour is still occurring open a new ticket


Support displaying Bayer formats applying appropriate demosaicing algorithm.

Is your feature request related to a problem? Please describe.
As of June 2018, Harvester GUI supports displaying raw Bayer formats but doesn't support applying the appropriate demosaicing algorithm.
Describe the solution you'd like
Bayer formats are very popular today so Harvester GUI should also support it.
It'd be nice if users can switch on or off applying the demosaicing algorithm.
I assume it can be done by implementing the demosaicing algorithm in OpenGL shading language. This should be excel at the performance point of view.
Describe alternatives you've considered
No alternatives.
Additional context
No additional context.

This should be done by the Harvester GUI.


[TASK] Address "switch" should support switching to a different wallet

Overview
Currently the "switch" functionality when clicking the wallet button only allows you to switch your metamask address or your polkadotJS address. The flow for switching wallets entirely requires first disconnecting and then connecting a different wallet. This flow could be wrapped up together from "switch" as well, no?

I believe we should maintain the current functionality of switching accounts but rename it to switch account instead of just switch to reduce confusion. Additionally, we can add another button for switching to another wallet in the dropdown menu.

Yes, I think it makes sense to keep the current functionality, BUT improve the UI to make it more smooth/easy to use.


ZStream.empty make operation is stop and not return a actual retult.

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

for {
  _ <- if(true) ZStream.empty else ZStream.fromxx() // Incorrect usage
  r <- ZStream.fromxx()
}

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

@IceMimosa

1.当前仅当流应该就地终止，才应该使用 {} when (condition) 否则应该使用 if else


Add Python versions to GHA Windows

Signed-off-by: Gábor Lipták<EMAIL_ADDRESS>Win 3.7 fails with
FAIL: pyftpdlib.test.test_filesystems.TestAbstractedFS.test_validpath_external_symlink
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\pyftpdlib-1.5.8-py3.7.egg\pyftpdlib\test\test_filesystems.py", line 195, in test_validpath_external_symlink
    self.assertFalse(fs.validpath(u(testfn)))
AssertionError: True is not false

I think there's no need to test so many python versions on Windows.
2.7 and one (any) one recent 3.X is enough.
Same goes for other platforms except Linux.


Failed to load the linter-ruby package

[Enter steps to reproduce below:]

right click on an SVG file, open with Atom

Atom Version: 1.2.4
System: Mac OS X 10.10.5
Thrown From: Atom Core
Stack Trace
Failed to load the linter-ruby package
At ENOENT: no such file or directory, lstat '/Users/ryan/Library/Application Support/com.github.atom.ShipIt/update.pDrT760'

Error: ENOENT: no such file or directory, lstat '/Users/ryan/Library/Application Support/com.github.atom.ShipIt/update.pDrT760'
    at Error (native)
    at fs.lstatSync (fs.js:844:18)
    at Object.fs.lstatSync (ATOM_SHELL_ASAR.js:189:16)
    at realpathSync (fs.js:1454:21)
    at Object.fs.realpathSync (ATOM_SHELL_ASAR.js:272:24)
    at toRealPath (module.js:109:13)
    at tryFile (module.js:105:22)
    at tryExtensions (module.js:115:20)
    at Function.Module._findPath (module.js:156:20)
    at Module._resolveFilename (module.js:334:25)

Commands

Config
{
  "core": {
    "themes": [
      "atom-dark-ui",
      "atom-dark-syntax"
    ]
  },
  "editor": {
    "invisibles": {}
  }
}

Installed Packages
# User
No installed packages

# Dev
No dev packages

Duplicate of #9559 and #7399, please subscribe there for updates.


MAINT, BENCH: would be good to be able to run benchmarks with raw asv + virtualenv reliably

Checkout latest main branch and use latest stable release of asv (0.6.3.). Switch to virtualenv config (because that's what I want to use, and it should work):
--- a/benchmarks/asv.conf.json
+++ b/benchmarks/asv.conf.json
@@ -57,8 +57,8 @@
     // If missing or the empty string, the tool will be automatically
     // determined by looking for tools on the PATH environment
     // variable.
-    // "environment_type": "virtualenv",
-    "environment_type": "mamba",
+    "environment_type": "virtualenv",
+    // "environment_type": "mamba",
     "build_cache_size": 10,

Run the benchmarks to assess performance delta in a PR:
asv continuous -e -b "SingleDist.*" main enh_20561
Fail with No module named build

Couldn't load asv.plugins._mamba_helpers because
No module named 'libmambapy'
· Creating environments
· Discovering benchmarks
·· Uninstalling from virtualenv-py3.11-Cython-meson-python-numpy-pybind11-pytest-pythran
·· Building 91770f5c <enh_20561> for virtualenv-py3.11-Cython-meson-python-numpy-pybind11-pytest-pythran
·· Error running /home/treddy/github_projects/scipy/benchmarks/env/707a4b5825152af35803cd1f32741469/bin/python -m build (exit status 1)
   STDOUT -------->

   STDERR -------->
   /home/treddy/github_projects/scipy/benchmarks/env/707a4b5825152af35803cd1f32741469/bin/python: No module named build

·· Failed: trying different commit/environment
·· Uninstalling from virtualenv-py3.11-Cython-meson-python-numpy-pybind11-pytest-pythran
·· Building 8acedb1b <main> for virtualenv-py3.11-Cython-meson-python-numpy-pybind11-pytest-pythran
·· Error running /home/treddy/github_projects/scipy/benchmarks/env/707a4b5825152af35803cd1f32741469/bin/python -m build (exit status 1)
   STDOUT -------->

   STDERR -------->
   /home/treddy/github_projects/scipy/benchmarks/env/707a4b5825152af35803cd1f32741469/bin/python: No module named build

·· Failed to build the project and import the benchmark suite.

I'm not keen on using conda/mamba for this if I can stick with what I'm used to. I can circumvent that problem by fixing what looks like out of date entry in the dep matrix:
--- a/benchmarks/asv.conf.json
+++ b/benchmarks/asv.conf.json
@@ -41,7 +41,7 @@
         "pytest": [],
         "pythran": [],
         "pybind11": [],
-        "meson-python": [],
+        "meson": [],
     },

but it still fails, presumably for another reason that needs investigation:
Couldn't load asv.plugins._mamba_helpers because
No module named 'libmambapy'
· Creating environments.....
· Discovering benchmarks.......
·· Uninstalling from virtualenv-py3.11-Cython-meson-numpy-pybind11-pytest-pythran
·· Building 91770f5c <enh_20561> for virtualenv-py3.11-Cython-meson-numpy-pybind11-pytest-pythran.............................................
·· Installing 91770f5c <enh_20561> into virtualenv-py3.11-Cython-meson-numpy-pybind11-pytest-pythran
·· Failed: trying different commit/environment

@HaoZeke this is the diff I currently need to run with virtualenv instead of mamba for environment_type. This is similar to reports from https://github.com/airspeed-velocity/asv/issues/1389. Should I try to make a change like this in SciPy config, or is there something cleaner I could do?
diff --git a/benchmarks/asv.conf.json b/benchmarks/asv.conf.json
index 0247b4581..0dc8515c7 100644
--- a/benchmarks/asv.conf.json
+++ b/benchmarks/asv.conf.json
@@ -20,9 +20,9 @@
     //
     // "install_command": ["in-dir={env_dir} python -mpip install {wheel_file}"],
     // "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"],
-    // "build_command": [
-    //     "PIP_NO_BUILD_ISOLATION=false python -m pip install . --no-deps --no-index -w {build_cache_dir} {build_dir}"
-    // ],
+    "build_command": [
+        "PIP_NO_BUILD_ISOLATION=false python -m pip wheel . --no-deps --no-index -w {build_cache_dir} {build_dir}"
+    ],

     // The base URL to "how a commit for the project.

Thanks for documenting this change, Tyler. I had to do the same thing on my machine in order to use virtualenv.
I think we should update the Scipy config, and add a comment explaining that this build_command should be uncommented if using virtualenv.


Add upgrade helptext

Questions
Answers

Description?
Adds small hint about debug mode.

Type?
improvement

BC breaks?
no

Deprecations?
no

Fixed ticket?

Sponsor company

How to test?
No need to test, merge right away.

👍

thanks @Hlavtox


Please make pytest optional

Please make pytest optional, and other small things. See attached patch.
Patch.patch.txt

Yaru, at some point we need to align this plugin more towards the latest version of our example plugins. I think Lauri is preparing something new, but if this is not the case we can just update it to match the current example here: https://github.com/nomad-coe/nomad-schema-plugin-example

ok! @Pepe-Marquez


install watch as part of demo run

Please describe the Improvement and/or Feature Request
On a new machine that didn't have watch installed and needed it to run the demo. Would be great to install this as part of the demo or a script to install requisites or as part of a make target.
Scope (please mark with X where applicable)

New Functionality          [ ]
Install                    [ ]
SMI Traffic Access Policy  [ ]
SMI Traffic Specs Policy   [ ]
SMI Traffic Split Policy   [ ]
Permissive Traffic Policy  [ ]
Ingress                    [ ]
Egress                     [ ]
Envoy Control Plane        [ ]
CLI Tool                   [ ]
Metrics                    [ ]
Certificate Management     [ ]
Sidecar Injection          [ ]
Logging                    [ ]
Debugging                  [ ]
Tests                      [X ]
CI System                  [ ]
Project Release            [ ]

Possible use cases
As a user, I want to run the demo as smoothly as possible.

Installing packages is one of those things that is both very distro-specific and personal preference that I'd be afraid to maintain a script here to do it automatically. Since it's already documented as a pre-req, the most I would want to do here ideally would be point to the pre-req docs if it's not installed. If it's critical to have installed beforehand, I would suggest we download a static binary to bin/ and invoke that if it's not already installed. I'm not sure that's worth the hassle though for what it does.
Perhaps an additional echo "The demo is up and running!" or some other confirmation message right before the watch is called would help clarify that even if the watch fails, the rest of the script still worked.

This seems unnecessary to include as a part of the demo run. I agree with what @nojnhuh mentioned in https://github.com/openservicemesh/osm/issues/1785#issuecomment-707975728.
Since this is clearly mentioned as a prerequisite in the demo guide https://github.com/openservicemesh/osm/blob/main/demo/README.md#system-requirements, I am closing this one.
Feel free to reopen if you disagree.


[FLINK-15649] Support mounting volumes

What is the purpose of the change
This pull request let user can mount customized volume to JobManager or TaskManager pod.
Brief change log
And the MountVolumesDecorator set the Volume and the VolumeMount to the given pod and container.
Verifying this change
This change added tests and can be verified as follows:

Added test that validates MountVolumesDecorator mounting volume correctly.

Does this pull request potentially affect one of the following parts:

The public API, i.e., is any changed class annotated with @Public(Evolving): no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented?  JavaDocs

I assume we can close this PR, because it is subsumed by FLINK-15656.

Closing the PR because it seems to be subsumed by #14629.


VM fails to boot correctly with additional drives via virtualisation.qemu.drives

Describe the bug
NixOS fails to boot when an additional drive is attached. It successfully boots when the vm script is copied and the extra -drive parameter is removed.
Stage 1 completes successfully.  Stage 2 fails. The earliest sign something is wrong are messages from systemd during very early boot of the form: [...]systemd-integritysetup-generator failed with exit status 127..
<<< NixOS Stage 2 >>>

[  108.986761] EXT4-fs (vda): re-mounted. Quota mode: none.
[  108.988017] booting system configuration /nix/store/6bdyci4yddk6khaq0r3yfn1idxl5if5c-nixos-system-nose-23.05.20230421.2362848
running activation script...
[  109.053320] stage-2-init: running activation script...
setting up /etc...[  109.469482] stage-2-init: setting up /etc...

starting systemd...
[  110.821499] systemd[1]: systemd 253.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
[  110.822543] systemd[1]: Detected virtualization qemu.
[  110.822700] systemd[1]: Detected architecture arm64.

Welcome to NixOS 23.05 (Stoat)!

[  110.889698] systemd[1]: bpf-lsm: Failed to link program; assuming BPF LSM is not available
[  110.974460] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator failed with exit status 127.
[  110.977978] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 127.
[  110.980356] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-cryptsetup-generator failed with exit status 127.
[  111.001806] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-run-generator failed with exit status 127.
[  111.002271] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-hibernate-resume-generator failed with exit status 127.
[  111.002753] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-system-update-generator failed with exit status 127.
[  111.003229] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-debug-generator failed with exit status 127.
[  111.003687] (sd-execu[610]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-bless-boot-generator failed with exit status 127.

Later, services start failing, starting with:
[    3.184649] systemd[1]: Failed to start Load Kernel Modules.
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.

Context:
I wish to create a nixos VM to run on my aarch64-darwin host. I want to attach additional storage via something like:
          virtualisation.qemu.drives = nixpkgs.lib.mkAfter [
            { file = "/Users/peterwaller-arm/nix/home.qcow2"; }
          ];

... so that my home directory storage can be managed in a separate volume from the nixos installation.
I am using nix run github:NixOS/nixpkgs/nixpkgs-unstable#darwin.builder to create a darwin builder, and then use the following flake:

(Click to expand) flake.nix
{
  outputs = { self, nixpkgs }: {
    packages.aarch64-darwin.default = (nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        "${nixpkgs}/nixos/modules/virtualisation/qemu-vm.nix"
        {
          virtualisation.host.pkgs = nixpkgs.legacyPackages.aarch64-darwin;
          virtualisation.graphics = false;
          virtualisation.cores = 8;
          virtualisation.memorySize = 4096;
          virtualisation.qemu.drives = nixpkgs.lib.mkAfter [
            {
              file = "/Users/peterwaller-arm/nix/home.qcow2";
            }
          ];
        }
        {
          system.stateVersion = "23.05";
          networking.hostName = "nixos-test";
          services.getty.autologinUser = "root";
        }
      ];
    }).config.system.build.vm;
  };
}

Steps To Reproduce
Steps to reproduce the behavior:

Build a VM with the darwin.builder configured.
Set virtualisation.qemu.drives = nixpkgs.lib.mkAfter [ to some local path.
Use qemu-img create to create the image at that path.
Try to boot and see it fail.

Expected behavior
The machine should boot happily.
Notify maintainers
(not sure who to CC, run out of time to dig, will try to dig later) @YorikSar  @roberth @domenkozar

Hmm. Not sure if this is a bug or not, though it is an unfortunate failure mode at least. I've come to realise that NixOS's vm run wrapper doesn't quite fit the way of using it I had in mind.
I had missed that by default, the store is being mounted with 9p from the host. When it's failing it's behaving a bit as though If I set:
virtualisation.useNixStoreImage = true;
virtualisation.writableStore = true;
virtualisation.writableStoreUseTmpfs = false;

Then everything works, even with extra disks. So I suspect something is going wrong with the 9p store when the extra disks are present.
One slightly unfortunate thing: for my use case I wanted to build an image (aarch64-linux) development machine on my (aarch64-darwin) host, and then once that image is built, continue to develop it from within the guest for months at a time (before eventually perhaps rebuilding from scratch on the host).
At the moment I see that with writableStore set, the nix store ends up as a $TMPDIR/store.img file which is discarded each time the guest is booted so that changes are lost. I wanted to have a volume for my home directory and the OS separately so that I may discard the host OS every so-often. I'd quite like to be able to avoid overwriting that on every boot.
I include the full log in the details block below in case it's of use to someone to search for error messages.

[    0.104707] armv8-pmu pmu: hw perfevents: failed to probe PMU!

<<< NixOS Stage 1 >>>

loading module virtio_balloon...
loading module virtio_console...
loading module virtio_rng...
loading module dm_mod...
running udev...
Starting systemd-udevd version 253.2
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
starting device mapper and LVM...
checking /dev/vda...
fsck (busybox 1.36.0)
[fsck.ext4 (1) -- /mnt-root/] fsck.ext4 -a /dev/vda
/dev/vda: clean, 3936/65536 files, 130670/262144 blocks
mounting /dev/vda on /...
mounting nix-store on /nix/.ro-store...
mounting tmpfs on /nix/.rw-store...
mounting shared on /tmp/shared...
mounting xchg on /tmp/xchg...
mounting overlay filesystem on /nix/store...

<<< NixOS Stage 2 >>>

running activation script...
setting up /etc...
starting systemd...

Welcome to NixOS 23.05 (Stoat)!

[    2.737130] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 127.
[    2.738449] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-hibernate-resume-generator failed with exit status 127.
[    2.775572] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 127.
[    2.776038] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-getty-generator failed with exit status 127.
[    2.776544] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-bless-boot-generator failed with exit status 127.
[    2.777013] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-veritysetup-generator failed with exit status 127.
[    2.777543] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator failed with exit status 127.
[    2.778030] (sd-execu[591]: /nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-run-generator failed with exit status 127.
[  OK  ] Created slice Slice /system/getty.
[  OK  ] Created slice Slice /system/modprobe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[  OK  ] Started Forward Password R…uests to Wall Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target Containers.
[  OK  ] Reached target Path Units.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slice Units.
[  OK  ] Reached target Swaps.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on Userspace Out-Of-Memory (OOM) Killer Socket.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Listening on udev Kernel Socket.
         Mounting Huge Pages File System...
         Mounting POSIX Message Queue File System...
         Mounting Kernel Debug File System...
         Starting Create List of Static Device Nodes...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module drm...
         Starting Load Kernel Module efi_pstore...
         Starting Load Kernel Module fuse...
         Starting mount-pstore.service...
         Starting Journal Service...
         Starting Load Kernel Modules...
         Starting Load/Save OS Random Seed...
         Starting Coldplug All udev Devices...
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Finished Create List of Static Device Nodes.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Load Kernel Module configfs.
[  OK  ] Finished Load Kernel Module efi_pstore.
         Mounting Kernel Configuration File System...
[    3.505884] (store.sh)[620]: mount-pstore.service: Failed to execute /nix/store/q71g8qphwwl5mcwg0zyzwpqpz8rg23nw-mount-pstore.sh: Too many open files
[    3.508538] systemd[1]: Failed to mount Kernel Configuration File System.
[FAILED] Failed to mount Kernel Configuration File System.
See 'systemctl status sys-kernel-config.mount' for details.
[    3.516998] systemd[1]: Failed to start mount-pstore.service.
[FAILED] Failed to start mount-pstore.service.
See 'systemctl status mount-pstore.service' for details.
[  OK  ] Finished Load Kernel Module fuse.
         Mounting FUSE Control File System...
[  OK  ] Finished Load Kernel Module drm.
[    3.554385] systemd[1]: Failed to start Journal Service.
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[    3.556877] systemd[1]: Failed to start Load Kernel Modules.
[FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details.
[    3.560231] systemd[1]: Failed to start Load/Save OS Random Seed.
[FAILED] Failed to start Load/Save OS Random Seed.
See 'systemctl status systemd-random-seed.service' for details.
[  OK  ] Finished Coldplug All udev Devices.
[  OK  ] Mounted FUSE Control File System.
         Starting Firewall...
[  OK  ] Stopped Journal Service.
         Starting Journal Service...
         Starting Apply Kernel Variables...
[  OK  ] Finished Create Static Device Nodes in /dev.
         Starting Rule-based Manage…for Device Events and Files...
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Finished Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Userspace Out-Of-Memory (OOM) Killer...
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Started Rule-based Manager for Device Events and Files.
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  OK  ] Reached target System Initialization.
[  OK  ] Started logrotate.timer.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timer Units.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Listening on Nix Daemon Socket.
[  OK  ] Reached target Socket Units.
[  OK  ] Reached target Basic System.
         Starting Kernel Auditing...
         Starting DHCP Client...
         Starting Logrotate configuration check...
         Starting Name Service Cache Daemon (nsncd)...
[  OK  ] Started Reset console on configuration changes.
         Starting resolvconf update...
[  OK  ] Started Userspace Out-Of-Memory (OOM) Killer.
         Starting D-Bus System Message Bus...
[FAILED] Failed to start D-Bus System Message Bus.
See 'systemctl status dbus.service' for details.
[FAILED] Failed to start Logrotate configuration check.
See 'systemctl status logrotate-checkconf.service' for details.
         Starting D-Bus System Message Bus...
[FAILED] Failed to start Firewall.
See 'systemctl status firewall.service' for details.
[  OK  ] Started Name Service Cache Daemon (nsncd).
[  OK  ] Finished Kernel Auditing.
[FAILED] Failed to start resolvconf update.
See 'systemctl status resolvconf.service' for details.
[  OK  ] Reached target Preparation for Network.
[  OK  ] Reached target All Network Interfaces (deprecated).
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Reached target User and Group Name Lookups.
         Starting Networking Setup...
         Starting User Login Management...
[FAILED] Failed to start D-Bus System Message Bus.
See 'systemctl status dbus.service' for details.
         Starting D-Bus System Message Bus...
[FAILED] Failed to start D-Bus System Message Bus.
See 'systemctl status dbus.service' for details.
[FAILED] Failed to start User Login Management.
See 'systemctl status systemd-logind.service' for details.
[FAILED] Failed to start Networking Setup.
See 'systemctl status network-setup.service' for details.
[DEPEND] Dependency failed for Extra networking commands..
[  OK  ] Reached target Network.
         Starting D-Bus System Message Bus...
         Starting SSH Daemon...
[  OK  ] Stopped User Login Management.
         Starting Load Kernel Module drm...
         Starting Permit User Sessions...
[  OK  ] Finished Load Kernel Module drm.
[FAILED] Failed to start SSH Daemon.
See 'systemctl status sshd.service' for details.
[FAILED] Failed to start D-Bus System Message Bus.
See 'systemctl status dbus.service' for details.
[FAILED] Failed to start Permit User Sessions.
See 'systemctl status systemd-user-sessions.service' for details.
[  OK  ] Listening on Load/Save RF …itch Status /dev/rfkill Watch.
         Starting D-Bus System Message Bus...
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
         Starting User Login Management...
[FAILED] Failed to start D-Bus System Message Bus.
See 'systemctl status dbus.service' for details.
[FAILED] Failed to start D-Bus System Message Bus.
See 'systemctl status dbus.service' for details.
[FAILED] Failed to start User Login Management.
See 'systemctl status systemd-logind.service' for details.
[  OK  ] Stopped User Login Management.
[  OK  ] Listening on D-Bus System Message Bus Socket.
         Starting Load Kernel Module drm...
[  OK  ] Finished Load Kernel Module drm.
         Starting User Login Management...
[  OK  ] Stopped SSH Daemon.
         Starting SSH Daemon...
[FAILED] Failed to start User Login Management.
See 'systemctl status systemd-logind.service' for details.
[  OK  ] Stopped User Login Management.
         Starting Load Kernel Module drm...
[  OK  ] Started SSH Daemon.
[  OK  ] Finished Load Kernel Module drm.
         Starting User Login Management...
[FAILED] Failed to start User Login Management.
See 'systemctl status systemd-logind.service' for details.
[  OK  ] Stopped User Login Management.
         Starting Load Kernel Module drm...
[  OK  ] Finished Load Kernel Module drm.
         Starting User Login Management...
[FAILED] Failed to start User Login Management.
See 'systemctl status systemd-logind.service' for details.
[  OK  ] Stopped User Login Management.
         Starting Load Kernel Module drm...
[  OK  ] Finished Load Kernel Module drm.
[FAILED] Failed to start User Login Management.
See 'systemctl status systemd-logind.service' for details.
[  OK  ] Started DHCP Client.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target Multi-User System.

At the moment I see that with writableStore set, the nix store ends up as a $TMPDIR/store.img file which is discarded each time the guest is booted so that changes are lost.

This is strange. writableStoreUseTmpfs should have prevented that. Did you delete local nixos.qcow2 image before you tried running with this flag? It sometimes persists things that prevent major changes in VM configuration.
As for the title issue, I didn't try this configuration with attached drives. I  would guess there is some assumption in qemu-vm module that first drive is treated specially or something like that.

Hi there,
Can you check that /mnt-root/nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator references (or is) a binary for the correct architecture? file, etc. (or even better: try to run the binaries yourself).

One thing is that nix on the host is very slow with a possible IO/filesystem bottleneck among other things (antivirus I don't have control over?).

Hm, alright but:

Within the VM everything is blazing fast. I want to build a VM on the host once and then work more or less within the VM only.

Does that mean you want to run without the host nix store? In that case, it just makes sense to have a Nix store image or have it baked into your "whole system partition" (enabled by useBootLoader = true;)

Can you check that /mnt-root/nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator references (or is) a binary for the correct architecture? file, etc. (or even better: try to run the binaries yourself).

The file looks right.
/nix/store/vjn0b3lxayixaz41vwvn76dl5f83p1x5-systemd-253.3/lib/systemd/system-generators/systemd-cryptsetup-generator:
ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/2fcdcxfnwz1h8hvbxmggzk0c7vm1j2bn-glibc-2.37-8/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped

Does that mean you want to run without the host nix store? In that case, it just makes sense to have a Nix store image or have it baked into your "whole system partition" (enabled by useBootLoader = true;)

Yes, I'd like to keep the store within the image (and have it survive reboots). Unfortunately, I can't build a useBootLoader = true image, that results in the following failure.
My configuration has the darwin.builder image running and configured as a substituter on the macOS /etc/nix/nix.conf so that it is able to build things requiring a linux builder. Unfortunately this appears to introduce the extra constraint of requiring KVM.
Failed to find a machine for remote build!
derivation: 26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv
required (system, features): (aarch64-linux, [kvm])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([aarch64-linux], 8, [], [])
error: a 'aarch64-linux' with features {kvm} is required to build '/nix/store/26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

I'm also looking at writableStore:
https://github.com/NixOS/nixpkgs/blob/63b758df7bec553f89a2578f7b85e8d3da02ea50/nixos/modules/virtualisation/qemu-vm.nix#L144-L148
And I'm wondering really how I would want this to look: for my use case I don't want a case where if the nix host store is lost/garbage collected, the VM stops working because it expects the backing image to be available and it isn't anymore.
What I want really is to have the whole VM's state (store, root, home directory etc) live in my local directory, but to initialize that state copying from the store when I first run the VM. That way, when I have to reboot for host updates, when I come back and boot my VM again, it will be there so long as I have those qcow2 files.
Does such a feature seem reasonable to include in this VM machinery in principle, or do I need to look at using an alternative method?

Can you check that /mnt-root/nix/store/vs51yw4s7xmsqxivap64ry2yfzjhdxji-systemd-253.2/lib/systemd/system-generators/systemd-integritysetup-generator references (or is) a binary for the correct architecture? file, etc. (or even better: try to run the binaries yourself).

The file looks right.
/nix/store/vjn0b3lxayixaz41vwvn76dl5f83p1x5-systemd-253.3/lib/systemd/system-generators/systemd-cryptsetup-generator:
ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/2fcdcxfnwz1h8hvbxmggzk0c7vm1j2bn-glibc-2.37-8/lib/ld-linux-aarch64.so.1, for GNU/Linux 3.10.0, not stripped

Does that mean you want to run without the host nix store? In that case, it just makes sense to have a Nix store image or have it baked into your "whole system partition" (enabled by useBootLoader = true;)

Yes, I'd like to keep the store within the image (and have it survive reboots). Unfortunately, I can't build a useBootLoader = true image, that results in the following failure.
My configuration has the darwin.builder image running and configured as a substituter on the macOS /etc/nix/nix.conf so that it is able to build things requiring a linux builder. Unfortunately this appears to introduce the extra constraint of requiring KVM.
Failed to find a machine for remote build!
derivation: 26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv
required (system, features): (aarch64-linux, [kvm])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([aarch64-linux], 8, [], [])
error: a 'aarch64-linux' with features {kvm} is required to build '/nix/store/26p70ycm60ychxj98pa358yk39fw7z45-nixos-disk-image.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

Indeed, you need KVM to run a Linux VM for creating the disk image. That's an unfortunate limitation of make-disk-image, it can be reduced but require serious efforts to get rid of that.

Looks like the option I want is: "use a nix store image", but with the additional effect of it being a persistent writable copy of the store-as-it-was-built, rather than "a CoW qcow2 image backed by an image in /nix/store which gets overwritten on every VM boot".

so persistNixStoreImage I guess?

Something like that sounds excellent ❤

I note that we have detracted from the original issue as filed, which was that something is going wrong with the nix store. I'm happy to try and help track that down if you see value in it have any other ideas for diagnostics, even though I may not be a user of it (given that useNixStoreImage = writableStore = true; renders it a moot point).

Two logs for you. Captured with QEMU_KERNEL_PARAMS='loglevel=10  systemd.log_level=debug systemd.log_target=console boot.trace' USE_TMPDIR=0 TMPDIR=$PWD result/bin/run-nose-vm  &> trace-working.log and terminating after a manual count of ten with ctrl-a x.

trace-broken.log
trace-working.log

What I want really is to have the whole VM's state (store, root, home directory etc) live in my local directory, but to initialize that state copying from the store when I first run the VM.

This is basically already possible with
virtualisation.useNixStoreImage = true;
virtualisation.writableStore = true;
virtualusation.writableStoreUseTmpfs = false;

Your "root" disk (usually nixos.qcow2) whose location you can configure with virtualisation.diskImage then contains all the state including the store paths you have created inside the VM. Whenever the inputs to your VM change (e.g. when you update your flake or your channel), the Nix store image is rebuilt and used instead of the old one.
With the changes from https://github.com/NixOS/nixpkgs/pull/241373 this would change slightly where on every start of the VM you get a freshly built Nix store image.

where on every start of the VM you get a freshly built Nix store image.

@nikstur I appreciate the improvements you're making in that PR, but it would be good if it were possible to avoid throwing away the store state every boot as well. I still want to use the resulting machine as I would a normal machine, where I build some software into my home directory, have direnvs, and various things, and I don't want to have to rebuild those things from scratch every time I boot, and I don't want to have to rebuild the VM itself and reboot in order to arrange that this development artefacts appear inside the VM. Mostly I want to remain inside the VM since the host's I/O is much slower.

I don't want to have to rebuild those things from scratch every time I boot

With the options I showed, you don't have to rebuild any of those things. The Nix store image is mounted read-only (it has been this way ever since the option useNixStoreImage was introduced) and is made writable by layering an overlayfs over it. When virtualisation.writableStoreUseTmpfs is false, this writable upper directory that is layered over the read-only Nix store image is a persistent directory on the device that backs /. In the normal case this will be the disk configured with virtualisation.diskImage.  So with these settings, everything but the original store paths (those determined when you first built the VM) are stored in your persistent VM disk that you keep around for as long as you want. Only the "original" store paths are ever re-assembled into a Nix store image.
But maybe I'm misunderstanding what exactly you want a bit.

I don't want to have to rebuild the VM itself and reboot in order to arrange that this development artefacts appear inside the VM

This sounds like you want to share the home directory on your host with the one in your VM via something like 9p.

With the options I showed, you don't have to rebuild any of those things.

Ah, I had misunderstood, thanks; I had understood the store would be thrown away each time. I suppose the lower overlay directory being rebuilt each time isn't so bad assuming that is a rapid operation (and doesn't create a maintenance or skew issue should the base change accidentally, but that seems like it shouldn't be a problem.

This sounds like you want to share the home directory on your host with the one in your VM via something like 9p.

I'd love to do this if it were performant enough for my use cases, but it is not. I often interact with projects having hundreds of thousans of files, and the darwin host's I/O is a problem for this, introducing a lot of delay (seconds become minutes for some trivial operations). Inside the VM, linux handles it fine. That's how I've ended up with the 'work inside the VM and rebuild rarely pattern' which works very well for me.

I think you'll have a bit of trouble when you change the image, leading to disappeared store paths while they're still recorded as supposedly valid in the store db, as your store db is in the upper directory whereas the disappearing (because of a config / channel update) happens in the lower directory.
You might want to keep a link to the previous version around and use that as an extra store just for fixing up those discrepancies when they occur. Or you could just throw out the bad parts without actually recovering them, if that's ok for your use case.

Thanks @roberth, that's the sort of thing I was concerned about. I understand if I I may have to rebuild everything (and/or discard 'now bad' symlinks pointing into the store) should I change the base image. However, if the base is rebuilt each time but builds to exactly the same thing (e.g. because it's flake-locked and the lock doesn't change), that would be OK, or have I missed something?
In some ways it would be more convenient to me if we could avoid rebuilding the base erofs image every time on boot, for example by having a way of pointing the run-vm.sh script to a path (VM_BASE_IMAGE=$PWD/base.erofs?), and avoid rebuilding it if it already exists.

However, if the base is rebuilt each time but builds to exactly the same thing (e.g. because it's flake-locked and the lock doesn't change), that would be OK, or have I missed something?

If everything is locked, this should work fine.

In some ways it would be more convenient to me if we could avoid rebuilding the base erofs image every time on boot, for example by having a way of pointing the run-vm.sh script to a path (VM_BASE_IMAGE=$PWD/base.erofs?), and avoid rebuilding it if it already exists.

Mhm, I don't love it because that means you have to manually clean up state and it will never be clear when/that you have to do it. This already happens to me with the normal disk image of the VM.

I'll have to give the new way a go before expressing an opinion. The new way sounds OK so long as building the erofs is performant in my situation.


VoicemailProbe

android.intent.action.FETCH_VOICEMAIL
android.intent.action.NEW_VOICEMAIL

The intents get never caught by the receiver. Permissions are granted, still does not work. All solution where this worked online are at least 3 years old. What to do @llayman ?

Being more precise: when I send the broadcast intents via ADB they get caught and the VoicemailProbes are being collected, but the system test with the phone shows that those intents do not get sent.

@mmueller301 Perform the system test on API level 19 phone. If you cannot see the system-generated intents on API 19, then discard the probe.

Also on API 19 the intents are never sent. The probe will be discarded.


Groovy Builds fail due to out of date version on Mac OS X

All nightly builds failed with this error
https://ci.adoptopenjdk.net/view/Failing Builds/
WARNING: An illegal reflective access operation has occurred
04:31:54  WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/home/jenkins/workspace/build-scripts/jobs/jdk/jdk-linux-aarch64-openj9/pipelines/target/libs/adopt-shared-lib.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
04:31:54  WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
04:31:54  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
04:31:54  WARNING: All illegal access operations will be denied in a future release
 (edited)
9:28

This is really a duplicate of #1863 but I'll leave it since there is an active discussion thread ....

I've updated this issue to be about the Groovy concern.

We compile groovy using 2.5.6 here which I assume is causing the WARNING. This is less of a bug and more of an annoyance as the pipelines can run even when this warning occurs.
At least I don't think so as https://ci.adoptopenjdk.net/view/Failing Builds/job/build-scripts/job/jobs/job/jdk15/job/jdk15-mac-x64-openj9/5/consoleFull failed for seemingly no reason other than that error and some deprecation warnings as seen below
04:31:17  To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/6.5/userguide/gradle_daemon.html.
04:31:18  Daemon will be stopped at the end of the build stopping after processing
04:31:24  > Task :clean UP-TO-DATE
04:31:24  > Task :compileJava NO-SOURCE
04:31:30  > Task :compileGroovy
04:31:30  > Task :processResources NO-SOURCE
04:31:30  > Task :classes
04:31:33  > Task :uberjar
04:31:33
04:31:33  Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
04:31:33  Use '--warning-mode all' to show the individual deprecation warnings.
04:31:33  See https://docs.gradle.org/6.5/userguide/command_line_interface.html#sec:command_line_warnings
04:31:33
04:31:33  BUILD SUCCESSFUL in 20s
04:31:33  3 actionable tasks: 2 executed, 1 up-to-date
04:31:33  WARNING: An illegal reflective access operation has occurred
04:31:33  WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/pipelines/target/libs/adopt-shared-lib.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
04:31:33  WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
04:31:33  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
04:31:33  WARNING: All illegal access operations will be denied in a future release

The advice to avoid this sort of thing is upgrade Groovy to 3.x.x OR downgrade the java that groovy uses to 8


Update README.md

Minor typo spotted in the first paragraph

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[BUG]  TPCX-BB Query Failures probably due to  column selection failure in read_parquet for df with nulls

This impacts Quey-02, Query-03 ... and probably a lot more.
Query 2 Fail Trace
Traceback (most recent call last):
  File "/home/vjawa/tpcx-bb-2/tpcx_bb/xbb_tools/utils.py", line 280, in run_dask_cudf_query
    config=config,
  File "/home/vjawa/tpcx-bb-2/tpcx_bb/xbb_tools/utils.py", line 61, in benchmark
    result = func(*args, **kwargs)
  File "queries/q02/tpcx_bb_query_02.py", line 143, in main
    result_df = result_df.head(q02_limit)
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/dask/dataframe/core.py", line 1006, in head
    return self._head(n=n, npartitions=npartitions, compute=compute, safe=True)
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/dask/dataframe/core.py", line 1039, in _head
    result = result.compute()
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/dask/base.py", line 447, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/distributed/client.py", line 2693, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/distributed/client.py", line 1990, in gather
    asynchronous=asynchronous,
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/distributed/client.py", line 833, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/distributed/utils.py", line 339, in sync
    raise exc.with_traceback(tb)
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/distributed/utils.py", line 323, in f
    result[0] = yield future
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/distributed/client.py", line 1849, in _gather
    raise exception.with_traceback(traceback)
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py", line 276, in read_parquet_part
    dfs = [func(fs, rg, columns.copy(), index, **kwargs) for rg in part]
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py", line 276, in <listcomp>
    dfs = [func(fs, rg, columns.copy(), index, **kwargs) for rg in part]
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/dask_cudf/io/parquet.py", line 57, in read_partition
    **kwargs.get("read", {}),
  File "/home/vjawa/conda/envs/rapids-tpcx-bb-aug-19/lib/python3.7/site-packages/cudf/io/parquet.py", line 214, in read_parquet
    use_pandas_metadata=use_pandas_metadata,
  File "cudf/_lib/parquet.pyx", line 151, in cudf._lib.parquet.read_parquet
  File "cudf/_lib/parquet.pyx", line 201, in cudf._lib.parquet.read_parquet
  File "cudf/_lib/table.pyx", line 115, in cudf._lib.table.Table.from_unique_ptr
  File "cudf/_lib/column.pyx", line 411, in cudf._lib.column.Column.from_unique_ptr
RuntimeError: cuDF failure at: /opt/conda/envs/rapids/conda-bld/libcudf_1597882732206/work/cpp/src/column/column_view.cpp:56: Invalid null mask for non-zero null count.

Related CUDF issue: https://github.com/rapidsai/cudf/issues/6048

Do you have any regression info, when did you start seeing this issue?

Do you have any regression info, when did you start seeing this issue?

Not really.  The last sweep we have on file is on 0.15.0a200723 for 10k.  (the issue is not manifested on 1k)
Maybe @randerzander / @beckernick  / @ayushdg  , Have more recent sweeps that they can share.


feat: [PROD-9525] bold modified parameters

Before creating a PR, please check that:

[ ] Code is clean
[ ] Tests are still working (cypress tests and yarn test)
[ ] Changes don't cause new errors or warnings in browser console
[ ] Documentation is up-to-date
[ ] Breaking changes are clearly identified with conventional commits

Yes, it's possible to apply a class conditionally, good idea, thanks ! Is it better to introduce a new non required prop isDirty or we can pass it with all textFieldProps ?

Yes, it's possible to apply a class conditionally, good idea, thanks ! Is it better to introduce a new non required prop isDirty or we can pass it with all textFieldProps ?
@esasova a new non-required prop isDirty in all generic component would be nice :slightly_smiling_face:


HTML figures

have to be corrected

are now better, but somehow the in-between phase does not work

Is the version under posts/ the deployed version? Because there I can't see the pictures either? Or is that something else?

The deployed version is here
under _posts/... there is the source-file

Ah thanks!
I don't really see the bug, for me I can see all the pictures... Can you maybe describe it more?

Yeah it actually is not a bug anymore ( I fixed the bug).
Currently we only need to do two thins:

align the image of the tab view. The source files have different sizes -> the scale differently. However a single image spot should stay the same (this is almost perfect in the second tab view) -> so the change has to happen in the files: https://github.com/blogosfair/How-does-the-inductive-bias-influence-the-generalization-capability-of-neural-networks/blob/main/assets/img/2022-12-01-how-does-the-inductive-bias-influence-the-generalization-capability-of-neural-networks/Figure2.PNG and https://github.com/blogosfair/How-does-the-inductive-bias-influence-the-generalization-capability-of-neural-networks/blob/main/assets/img/2022-12-01-how-does-the-inductive-bias-influence-the-generalization-capability-of-neural-networks/Figure3.PNG
correct html such the height is flexible ( I will look into it, but I am not sure if this can be done in this markdown thing)


bug in single quotes

The single quotation marks are double quotes

Please read at least the top of README before opening new issues.


Practice Lab 6.1 – Cloud flows

Task 1.4 – Set status to rejected
In Dynamic content, search for outcome and select Outcome under the Get Outcome section.

Expand Show advanced options.

Select No for Approved.

Select Rejected for Status Reason.

Click Save

Note Popup that "This isn't the latest content".
Workaround
We selected "Overwrite other peoples changes" to move forward.

Added note stating the workaround.


Migrate to eclipse-temurin:8-jdk

Change to ubuntu based image which supports windows/amd64, linux/amd64, linix/armv/v7, linux/arm64/v8, linux/ppc64le

:wave: Thanks for the PR and sorry for just noticing this now!
I'll run through the tests in the coming days and see if we can get multi-platform support easily by using your PR.


Improve timelock code

Done while working on a timelock module. This is all the initial patches (except one) from https://github.com/rust-bitcoin/rust-miniscript/pull/408 (which is a PR displaying usage of the new timelock module).
Note, does not do the 'make TimelockInfo fields pub crate' change - I was unsure if this was correct.

Force push is rebase only.


Add a hint for enabling Rosetta on M* macs

Without Rosetta enabled on Macs the connection will not be established. So in prerequisites-section we should add the hint to install Rosetta by executing
softwareupdate --install-rosetta

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 61601aaf-6d5e-d56c-40d0-fcdbd698d62c
Version Independent ID: 774316b0-c7af-19d2-67a0-1f9faeec4bd2
Content: Configure Azure VPN Client - Azure AD authentication - macOS - Azure VPN Gateway
Content Source: articles/vpn-gateway/openvpn-azure-ad-client-mac.md
Service: vpn-gateway
GitHub Login: @cherylmc
Microsoft Alias: cherylmc

@devdeer-alex , thank you for your feedback. We'll review this and get back to you shortly!

@devdeer-alex , I checked this issue and found an internal work item which is being worked on by the Azure VPN Product Group team and they will update the docs soon to include the Rosetta requirement.
Since this is already in the works, we are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.

@GitaraniSharma-MSFT Thanks for the info!


The conversion of the message to JSON

The MissingStringifierException is thrown when StringifierRegistry does not contain the appropriate Stringifier. The message types which are not registered in the StringifierRegistry should be converted to JSON format to reduce the strictness of the Stringifier and StringifierRegistry usage.

Done: https://github.com/SpineEventEngine/core-java/pull/423


Support arguments to the exporter container such as --log-level

It would be great if you could add support for setting command-line flags when calling OpenCost. The currently available flags can be seen here: https://github.com/opencost/opencost/blob/3186c1eee8de0826e44f0b13d00bf06f86db787e/pkg/cmd/commands.go#L73-L75
This is useful for setting the log level of OpenCost when trying to identify a problem. For example, the current deployment can be editted with the following to set the log level to trace.
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: opencost
        args:
        - --log-level
        - trace

Sorry, I just realised that the same can be achieved by setting LOG_LEVEL in the environment.


chore(benchmark): remove custom allocators because they are flaky

Current dependencies on/for this PR:

main

PR #2004   👈

This stack of pull requests is managed by Graphite.


Bump x86_64 macOS build to use El Capitan

Homebrew 3.5.0 is targetted to drop Yosemite support, so let's drop it here too.
I've set up a temporary CI node to test a 10.11 cross-compile on 10.13 (using 10.11 dev tools: Xcode 7.3.1). Very much an experimental setup but we'll see how it works.
CI changes will be dropped from the PR and I'll do it properly in a subsequent PR.

10.11 cross-compile on 10.13 (using 10.11 dev tools: Xcode 7.3.1)

So the way I've done this is very much like a cross-compile that people would do for cross-archs or cross-systems. It sounds like overkill for doing so across macOS versions, particularly when the dev tools are the same, but I think it is significantly safer to do so as it should disable any runtime configure checks that base things on the current running OS.
One caveat of cross-compiling is that Ruby requires the host Ruby to be the same version, which it isn't here. Instead of compiling twice, I have put in a workaround to use the in-tree miniruby like the native build would. It's definitely a bit hacky but at least it doesn't require any inreplace calls.

OpenSSL build currently fails for unrelated reasons. Looks like a test certificate has expired the moment we've entered June. It affects upstream's CI too so there should be a patch there soon. Once that's fixed we should have a test bottle we can try.

@Bo98 Nice work here so far. Just FYI: I think given the amount of code changes we're requiring for this: unless this works with minimal/no more code changes needed by EOD today or tomorrow: we should just raise the minimum requirement to 10.12 instead.

brew tests passes with this on El Capitan, except some Cask quarantine tests, which are definitely a recent-ish regression but not likely to be Ruby related. So it looks like our cross-compile setup works!
I'm also going to run it against Ruby's test suite (or at least the parts which can run against an external Ruby install) but I reckon anything it flags there will be pre-existing.

Great work @Bo98!


Code broke with crossbeam-utils 0.8.10. Worked in 0.8.9. Not a ZST.

I discovered that my code was failing to build after running cargo update. The root cause is due to MR 834 and I think there's a teaching moment for me to understand why this change broke my code. Here is a minimal repro of code that compiles (don't run it, it will panic due to bounds checks) using AtomicCell:
use crossbeam_utils::atomic::AtomicCell;
use std::borrow::Cow;

#[derive(Clone)]
struct Data {
    // Don't be a ZST
    val: u8
}

fn fn_2<F>(f: F)
where
    F: Fn(i64) -> Result<(), ()> + Sync + Send {
}

fn get_data<'a>(data: &'a [AtomicCell<Cow<Data>>], i: usize) -> &'a Data {
    unsafe { data[i].as_ptr().as_ref().unwrap() }
}

fn main() {
    let data: Vec<AtomicCell<Cow<Data>>> = vec![];

    fn_2(|i| {
        data[0].store(Cow::Owned(Data { val: 0 }));

        let a = get_data(&data, 0);

        data[1].store(Cow::Borrowed(a));

        Ok(())
    });

    let _ = &data[0];
}

If you use crossbeam-utils = "=0.8.9" in your Cargo.toml, this code compiles (with warnings). If you update the version to =0.8.10, you get this error:
error[E0597]: `data` does not live long enough
  --> src/main.rs:24:9
   |
23 |     fn_2(|i| {
   |          --- value captured here
24 |         data[0].store(Cow::Owned(Data { val: 0 }));
   |         ^^^^ borrowed value does not live long enough
...
34 | }
   | -
   | |
   | `data` dropped here while still borrowed
   | borrow might be used here, when `data` is dropped and runs the `Drop` code for type `Vec`

The code is kinda sketchy, but the idea is some entries in the Vec are values while other entries are references to other values in the same Vec. While this approach is likely a bad idea (you can get dangling references when data is dropping) and I've since switched from Cow to Arc, I do want to know why compilation broke.
Reading the MR, there was discussion on whether to do a minor or major semver bump and that people using ZSTs would be broken. But the above repro doesn't use a ZST. I'm curious why this code breaks without using a zero sized type and whether this is actually a bug in the Rust compiler.

Sorry for the breakage! This is due to the addition of the Drop implementation to the AtomicCell. (See nomicon's this page for details)
We were aware of the possibility of breakage due to the Drop implementation, but I considered this a minor change based on the fact that the RFC1105 does not consider this to be a major change:

Adding Drop implementation is also a breaking change, but RFC1105 treats only the "fundamental" trait implementation as a major change and all others as minor changes (Drop is not fundamental trait).

Reading the MR, there was discussion on whether to do a minor or major semver bump and that people using ZSTs would be broken.

The main discussion in #834 was about DSTs (Dynamically Sized Types), not ZSTs (Zero Sized Types).

The drop check makes total sense, and the example in the Nomicon is eerily similar to what I was doing. You all did everything by the book, so c'est la vie. We'll adopt a different strategy on our end. The understanding is all I really wanted out of this.
Thanks!


[FLINK-1520]Read edges and vertices from CSV files

[FLINK-1520]Read edges and vertices from CSV files
Changes done->

Added a GraphCsvReader class which has 2 CsvReaders as members EdgeReader and VertexReader

To make smooth chaining of functions possible for configuration of the member CsvReaders implemented the configuration methods in CsvReader in GraphCsvReader so that all the configurations can be done on either Edge Csv Reader or Vertex Csv reader.
Since specifying types is necessary because of type-erasure, implemented a types method in the GraphCsvReader class that returns a Graph with the specified types as the type for VertexID, Vertex Value and Edge Value. Other way for doing this was sending the types in a method to construct the graph itself but to make it as similar to CsvReader as possible this approach was taken.

Added 3 methods in Graph.java similar to other methods for Graph creation. These methods use one mandatory path and one optional path and optional mapper for Graph Creation. Only difference is that these methods return an instance of GraphCsvReader instead of Graph.

3)Added appropriate methods in GraphCreationITCase and GraphCreationWithMapperITCase,java
Also added createTempFile() method to both to help with the tests.

Added the documentation for the new functionalities to gelly_guide.md

Closed the previous pull request and made a new one with a fresh branch because the previous changes are not merged yet.

Hey @shghatge,
this is a great first try, you got the logic right and I really like the detailed javadocs ^^
I left a few inline comments, which should be easy to fix.
Let me also elaborate a bit on some general guidelines:

Code formatting: we don't really have a strict Java code style, but there a few things you can improve. For your code to be readable, it is nice to leave a space after the commas separating arguments. For example myMethod(arg1, arg2, arg3) instead of myMethod(arg1,arg2,arg3).
We usually separate the closing of a parenthesis and the opening of a curly bracket with a space, i.e. myMethod() { ... } instead of  myMethod(){ ... }.
Also, try to avoid adding new lines if they are not needed.
Regarding the types missing, this is not creating an error, but gives a warning. You can turn on warning notification settings in your IDE to avoid this.

I like it that you added separate methods includeFields methodsfor vertices and edges. It would probably make sense to do the same for the rest of the methods. For example, you might want to skip the first line in the edges file, but not in the vertices file. Right now, you are forced to either do both or none. Alternatively, we could add parameters to the existing methods, to define the behavior for edges and vertices files separately. For examplepublic GraphCsvReader lineDelimiter(String VertexDelimiter, EdgeDelimiter)`. What do you think?

Finally, in order to catch issues like the one with the null VertexReader, you should always try to test as much functionality you have added as possible. In this case, it would be a good idea to add a test reading from edges only and some tests for the different methods you have added.

Let me know if you have questions!

Hello @vasia
I will follow the guidelines and add the tests that are suggested by you when making a commit.
For the separate configuration methods issue, I was thinking more along the lines that if we want to configure the readers separately, then we could use the get methods for the CsvReaders and then configure them. But I will add the separate method now.
Thanks for the detailed guidance.  :)

Updated PR

This looks very nice! Someone deserves a virtual :ice_cream: !
There are some tests missing:

test fromCSV with a Mapper
you just test types, ignoreFirstLines and ignoreComments; let's at least add tests for the lineDelimiter* and the fieldDelimiter* methods. I'm sure they work, but tests are written to guarantee that the functionality will also be there (at the same quality) in the future (i.e. some exotic code addition will not break it) :)

I saw an outdated Vasia comment on an unused import; always hit mvn verify before pushing - it would have caught that :D

Ah! And I just remembered! Maybe it makes sense to update the examples to use fromCSV when creating the Graph instead of getEdgesDataSet.

Hello @Andra
There is one test for fromCsv with mapper in GraphCreationWithMapperITCase.java
Should I add more tests for that?
Also for the examples comment, do you mean that I should update the Gelly guide by removing the examples for Csv file which use env.readCsvFile("");?
I will add the other tests. :)

Mea culpa. No the mapper test is fine;
For the examples comment, I meant to go through the classes in the example folder and to modify the way the graph is currently read. Right now, we fetch the edges via env.fromCsv and then use Graph.fromDataSet to create the graph. We should do it directly via Graph.fromCsv.
The example in the docs is fine, because it explains how fromDataSet works. That is still available.

Updated PR

Hi @shghatge ,
I left my set of comments inline. They are mostly related to coding style issues. I guess you should revisit the previous comments here.
Also, don't forget to rebase. It seems like there are some merge conflicts that need to be fixed :)

Updated PR

Nice and rebased. +1

Hi @shghatge! Thank you for the update :)
I left some comments inline. There are still some formatting issues in the code. Please, carefully go through your changes and try to be consistent. Also, there are still several warning regarding types, unused annotations, unused variables. Can you please try to remove them? Your IDE should have a setting that gives you the list of warnings.
Regarding the tests, better create new test files for your methods, since you need to test with files and currently other tests use collect().
Finally, I find the types() methods a bit confusing. Could we maybe have separate types methods for the vertices and edges? e.g. typesEdges(keyType, valueType), typesEdges(keyType), typesVertices(keyType, valueType) and typesVertices(keyType)?

Updated PR

Updated PR

Updated PR

Hi,
I just had a closer look at this PR and it made me seriously question the utility of a Graph.fromCSV method. Why? First of all because it's more limited than the regular env.fromCsv() in the sense that it does not allow POJOs and it would be a bit tedious to support that. There would be a need for methods with 2 to n fields, according to the amount of attributes present in the POJO.
Second, because, and I am speaking strictly as a user here, I would rather write:
private static DataSet<Edge<Long, Double>> getEdgesDataSet(ExecutionEnvironment env) {
	if(fileOutput) {
		return env.readCsvFile(edgeInputPath)
				.ignoreComments("#")
				.fieldDelimiter("\t")
				.lineDelimiter("\n")
				.types(Long.class, Long.class, Double.class)
				.map(new Tuple3ToEdgeMap<Long, Double>());
	} else {
		return CommunityDetectionData.getDefaultEdgeDataSet(env);
	}
}

than...
private static Graph<Long, Long, Double> getGraph(ExecutionEnvironment env) {
Graph<Long, Long, Double> graph;
if(!fileOutput) {
DataSet<Edge<Long, Double>> edges = CommunityDetectionData.getDefaultEdgeDataSet(env);
graph = Graph.fromDataSet(edges,
new MapFunction<Long, Long>() {
					public Long map(Long label) {
						return label;
					}
				}, env);
	} else {
		graph = Graph.fromCsvReader(edgeInputPath,new MapFunction<Long, Long>() {
			public Long map(Long label) {
				return label;
			}
		}, env).ignoreCommentsEdges("#")
				.fieldDelimiterEdges("\t")
				.lineDelimiterEdges("\n")
				.typesEdges(Long.class, Double.class)
				.typesVertices(Long.class, Long.class);
	}
	return graph;
}

Maybe it's just a preference thing... but I believe it's at least worth a discussion. On the other hand, the utility of such a method should have been questioned from its early Jira days, so I guess that's my mistake.
I would like to hear your thoughts on this.
Thanks!

Hi @andralungu,
do you mean support for POJOs as vertex / edge values?
I guess that's a limitation we can't easily overcome, I agree.
Still though, a nicely designed fromCsv() method would simplify the common case.
As for the examples, I don't like what they currently look like in this PR either. However, that's not a problem of fromCsv(). The if-block can be easily simplified by changing getDefaultEdgeDataSet to getDefaultGraph. The else-block looks longer because of the mapper, which, in the current examples is in the main method.
What I think is quite problematic, is the types() methods. Ideally, we would have the following:

types(K) : no vertex value, no edge value
types(K, VV): no edge value
types(K, EV): no vertex value
types(K, VV, EV): both vertex and edge values are present
However, because of type erasure, we can't have both 2 and 3. The current implementation (having separate typesEdges and typesVertices) means that both should always be called, even if not necessary. Another way would be to give 2 and 3 different names... So far I haven't been able to come up with a nice solution. Ideas?

Hi @vasia,
I also saw the types issue, but I had a feeling that this is the way it was decided in the previous comment. I would rather have different names for 2 and 3 than to force a call to typeVertices if it's not needed.

I hadn't realized that they would both need to be called in my previous comment, my bad.
Any idea for decent method names? typesNoEdgeValue and typesNoVertexValue seem really ugly to me :S

Yes, but then you would have the following methods: types, typesNoEdgeValue, typesNoVertexValue and again types. So, even if it's not 100% needed I'd try to keep it consistent. We could also make it more graph-oriented (the name types was generic). The following is just an example:
1). keyType(K)
2). keyAndVertexTypes(K, VV)
3). keyAndEdgeTypes(K, EV)
4). keyVertexAndEdgeTypes(K, VV, EV)
With a nice documentation, I think I'd understand what these are for :)

Still an overkill I think... Could another way be to have only types(K, VV, EV) with all 3 arguments and expect NullValue if a value is missing?

Hmmm :-? but can you pass NullValue to tyes... it expects Something.class. Can it be overwritten without type erasure getting in the way?
Anyway... I will let @shghatge take over from here :)

yes, I mean NullValue.class :)
I'd like to know @shghatge's opinion, too!

The only problem with assuming NullValue if a value is missing is that we can't return NullValue in place of VV.
I mean to say Graph<K, VV, EV> in this VV or EV can't be NullValue. otherwise that was what I was originally going for.
Maybe since any of the other methods to create DataSet/Graph don't provide a method to give EdgeValue as NullValue and just expect the user to map it (at least that is what I saw), maybe we could just remove the functionality. I had only added it since many examples seemed to use it so I thought it would be nice to have that functionality.
In any case we can just keep one typesNullEdge method too because if they don't want that, they can use normal overloaded types, 3 arguments for no NullValue, 2 arguments for null vertex and 1 argument for null vertex and edge and just one method named typesNullEdge to tell that only edges have NullValue.

I see your point @shghatge.
However, I think naming just one method differently will be confusing..
If we're going to have custom method names, let's go with @andralungu's suggestion above and make sure we document these properly.
I would prefer a bit shorter method names though.
How about:
1). keyType(K)
2). vertexTypes(K, VV)
3). edgeTypes(K, EV)
4). types(K, VV, EV)
?

Hi @shghatge, @andralungu,
I built on the current state of this PR and made the proposed changes above, together with some styling changes. You can see the result on my local branch.
If you agree with the changes, I will also update the documentation and merge this.
Thank you!

Hi @vasia,
Not sure whether this comment was issued for me... Nevertheless I left some suggestions inline. All in all, it covers the problems discussed in the 73!!!!! comments here. You forgot to properly document the edgeTypes(K, EV), etc methods.

Saw this "I will also update the documentation " afterwards... Sorry!

Thanks for the comments @andralungu!
@shghatge, can you please close this PR? I will make the docs update and open a new one, which will include your work and my changes if that's OK with you. Thank you!

@vasia  It is fine with me.