Error checking for conflicts

Hi there,
I tried to install ixmp from source and got the error below. Prior to this attempt, I uninstalled ixmp from conda. Any ideas of how to fix this problem? Thanks!
H:\Things in C Drive\IIASA\Energy Group\MESSAGE\GitHub\ixmp>pip install .
Processing h:\things in c drive\iiasa\energy group\message\github\ixmp
Requirement already satisfied: JPype1>=0.6.2 in c:\users\hunt\appdata\local\cont
inuum\anaconda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (0.6.3)
Requirement already satisfied: pandas in c:\users\hunt\appdata\local\continuum\a
naconda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (0.23.4)
Requirement already satisfied: xlsxwriter in c:\users\hunt\appdata\local\continu
um\anaconda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (0.9.6)
Requirement already satisfied: xlrd in c:\users\hunt\appdata\local\continuum\ana
conda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (1.0.0)
Requirement already satisfied: numpy>=1.9.0 in c:\users\hunt\appdata\local\conti
nuum\anaconda3\lib\site-packages (from pandas->ixmp==0.1.3.post0.dev23) (1.15.4)
Requirement already satisfied: pytz>=2011k in c:\users\hunt\appdata\local\contin
uum\anaconda3\lib\site-packages (from pandas->ixmp==0.1.3.post0.dev23) (2016.10)
Requirement already satisfied: python-dateutil>=2.5.0 in c:\users\hunt\appdata\l
ocal\continuum\anaconda3\lib\site-packages (from pandas->ixmp==0.1.3.post0.dev23
) (2.6.0)
Requirement already satisfied: six>=1.5 in c:\users\hunt\appdata\local\continuum
\anaconda3\lib\site-packages (from python-dateutil>=2.5.0->pandas->ixmp==0.1.3.p
ost0.dev23) (1.10.0)
Building wheels for collected packages: ixmp
Building wheel for ixmp (setup.py) ... done
Stored in directory: C:\Users\hunt\AppData\Local\Temp\pip-ephem-wheel-cache-0h
whhx3j\wheels\fd\f1\5e\914b1324ef66c00c49912bcdc21453232d0e98f4b3ad9cd13c
Successfully built ixmp
Error checking for conflicts.
Traceback (most recent call last):
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init_.py", line 2897, in _dep_map
return self._dep_map
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2691, in getattr
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init_.py", line 2888, in _parsed_pkg_info
return self.pkg_info
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2691, in getattr
raise AttributeError(attr)
AttributeError: _pkg_info
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_i
nternal\commands\install.py", line 503, in _warn_about_conflicts
package_set, dep_info = check_install_conflicts(to_install)
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_i
nternal\operations\check.py", line 108, in check_install_conflicts
package_set, _ = create_package_set_from_installed()
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_i
nternal\operations\check.py", line 47, in create_package_set_from_installed
package_set[name] = PackageDetails(dist.version, dist.requires())
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2635, in requires
dm = self.dep_map
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2899, in _dep_map
self.__dep_map = self.compute_dependencies()
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2908, in _compute_dependencies
for req in self.parsed_pkg_info.get_all('Requires-Dist') or []:
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2890, in parsed_pkg_info
metadata = self.get_metadata(self.PKG_INFO)
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 1410, in get_metadata
value = self._get(self.fn(self.egg_info, name))
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 1522, in _get
with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: 'c:\users\hunt\appdat
a\roaming\python\python36\site-packages\ixmp-0+unknown.dist-info\METADATA'
Installing collected packages: ixmp
Found existing installation: ixmp 0+unknown
Could not install packages due to an EnvironmentError: [Errno 2] No such file or
directory: 'c:\users\hunt\appdata\roaming\python\python36\site-packages
\ixmp-0+unknown.dist-info\RECORD'
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' comm
and.
H:\Things in C Drive\IIASA\Energy Group\MESSAGE\GitHub\ixmp>

I managed to fix the issue.
impx was removed from conda, but not completely remove from python. By manually removing the existing ixmp libraries and running the installation again, it worked!
Thanks!


Issue with usage

Since it does not print to stdout, I would make -o a required option in order to run as it will still run but will not produce any output unless a output dir is specified.

NVM I stand corrected without a -o it creates pkt2flow.out, closing this ticket


PX4/ecl v1.9.0 release blockers

[ ] https://github.com/PX4/ecl/pull/578
[ ] https://github.com/PX4/ecl/pull/593
[ ] https://github.com/PX4/ecl/pull/505

@priseborough @RomanBapst @mhkabir @CarlOlsson please update this issue with any outstanding PX4/ecl changes or issues needed before a v1.9.0 release.

https://github.com/PX4/ecl/pull/578 has been merged
https://github.com/PX4/ecl/pull/593 is awaiting test evidence from the author, but is otherwise a low risk change.
https://github.com/PX4/ecl/pull/505 is a more complex change and without proper testing including the revised logic for handling of optical flow timeouts could introduce bugs I am not comfortable with it being included in a stable release without significant testing.

Thanks Paul.
@mhkabir @RomanBapst thoughts on testing https://github.com/PX4/ecl/pull/505 this week for release inclusion? Either way we should discuss an appropriate test plan.

@RomanBapst Please help with testing where possible.
@dagar Any updates?

Let's remove the optical flow PR from release. It will require more testing and work which I'm not able to contribute right now.

We've arranged to get @PX4/testflights optical flow hardware (pmw3901), but I don't know the realistic time frame.

Let's remove the optical flow PR from release. It will require more testing and work which I'm not able to contribute right now.

Sounds like we're bumping it for v1.9.0, but can we still try to get a test plan together for these items?

@jkflying has done extensive testing of this - could you please pitch in here? We're flying this daily in Zurich.

@mhkabir what about https://github.com/PX4/ecl/pull/593?

@jkflying has done extensive testing of this - could you please pitch in here? We're flying this daily in Zurich.

I've rebased https://github.com/PX4/ecl/pull/505 on current master. The branch ecl_PR-505 was created for testing PX4/Firmware side. https://github.com/PX4/Firmware/tree/ecl_PR-505
Binaries available here under artifacts. http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-compile/detail/ecl_PR-505/1/pipeline

I'll give this a flight tomorrow if the weather is ok, I don't forsee any issues caused by https://github.com/PX4/ecl/pull/505 though since the pmw3901 runs at ~40hz.

@dagar should I test the exact binaries? I usually increase tx/rx buffer sizes a bit because of our particular setup, but I can disable our log streaming, avoidance etc if it is important to use the exact binaries.

@jkflying no, it's just for convenience.

Flight with PMW3901 was good. No weird twitches, seemed smooth, did multiple takeoffs with optical flow straight from the grass, did a mission with avoidance enabled. Everything seemed fine. Merging is OK from my side.

@jkflying Can you post the logs on the ECL PR please?


Newer cargo/toolchain versions cause rebuilds of build.rs-using deps

I've recently upgraded Rust from 1.67.0 to 1.71.0, and noticed that in our "build workspace" derivation, some dependencies are being re-compiled, even though they were compiled in "build workspace deps only" derivation.
I confirmed that just switching between 1.67.0 and 1.71.0 fixes/introduces the problem.
It looks like only build.rs (and deps that depend on them) are being rebuilt, and from the initial debugging session I suspect something about tracking which deps require rebuild changed between these two toolchain (well... probably cargo) versions.
Notably the files restored in /build/source/target/ from zstd (I use the zstd mode) archive produced by deps-only have different timestamps than originals: they are all reset to the default unix timestamp, while the original ones had the original timestamps. I suspect something about how cargo tracks these changed.

1.68.0 is unaffected, and 1.69 is affected.

Ironically the change was done there to display reasons for rebuilds, but the reason for rebuilding affected crates is not being displayed.
https://github.com/rust-lang/cargo/pull/11407

Do you have a minimal repro I can take a look at?
What does cargo say about which files have changed specifically?

Do you have a minimal repro I can take a look at?

Will try to minimize when I find some time.

What does cargo say about which files have changed specifically?

I can't get it to tell me anything, even with -vv. It just prints the Compiling proc-macros2 etc., and executes the build script.

You can also try running it with CARGO_LOG=cargo::core::compiler::fingerprint=trace and see if that shows anything interesting

I created a quick-start example based project where I've added syn as a depdency (which brings also proc-macro2), and couldn't reprod, even bringing more and more little details of the affected project.
CARGO_LOG of affected vs example shows a important difference:
Affected:
[2023-08-23T18:58:43Z DEBUG cargo::core::compiler::fingerprint] old local fingerprints deps "/nix/store/8xlxixbffyyra5w4xv8fabyfcwynfdh0-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/proc-macro2-1.0.66" precalculated="1.0.66"

Not affected:
quick-start-build> [2023-08-23T19:18:54Z DEBUG cargo::core::compiler::fingerprint] new local fingerprints deps "/nix/store/wzs1f02h3xw6i0ql0a12r6kbla1savr2-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/proc-macro2-1.0.66"

which reading the code suggests that in the affected build cargo somehow lost the information proc-macro2 does output rerun-if-changed=build.rs every time and is thus 'new-style' custom build.

I see that it affects another projects, so maybe I'm not just crazy. I reported it on cargo issue tracker, maybe I can get some help.

Not doing .cargo copying in self.mkDummySrc didn't seem to change anything.

Getting rid of installCargoArtifactsMode = true; does not remove the problem.
Getting rid of any differences between args passed to buildDepsOnly and building workspace does not remove the problem.
(I'm basically stripping improvements/optimizations and see if any fixes it).

The problem did not happen in release build!

We have an extra ci profile (inheriting from dev), so I tried all 3, and for some reason release doesn't have the issue. Erasing the extra settings for these profiles in Cargo.toml doesn't change anything, so I don't think it's about the customizations itself.

I'm not sure where I'm supposed to set that profile. If I add --release to cargoExtraArgs, it complains about not being allowed to set it twice, so it seems like it's already getting set somewhere but I'm not actually sure where.

So I set CARGO_PROFILE = "release"; on the deps only build and one of the main builds, and it still rebuilt the build.rs-using dependencies

If you ever build your own cargo ... command you need to take care of $CARGO_PROFILE itself if you want it to be compatible with crane.

There's also the cargoWithProfile helper which will automatically inject --profile ... or --release based on the value of CARGO_PROFILE

@Radvendii you can set CARGO_LOG = "cargo::core::compiler::fingerprint=trace"; and look through the logs to see why the build is being invalidated

first i get a bunch of logs like
[2023-09-13T01:19:34Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /build/source/target/release/.fingerprint/nickel-lang-core-c6da6289b4897f33/lib-nickel-lang-core
[2023-09-13T01:19:34Z DEBUG cargo::core::compiler::fingerprint] max output mtime for "/nix/store/bhyli42jac57wn36n7sl24n14m3ryynz-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/ansi_term-0.12.1" is "/build/source/target/release/deps/libansi_term-0948ac557423c31e.rmeta" 1.000000000s
[2023-09-13T01:19:34Z DEBUG cargo::core::compiler::fingerprint] all paths up-to-date relative to "/build/source/target/release/.fingerprint/ansi_term-0948ac557423c31e/dep-lib-ansi_term" mtime=1.000000000s
[2023-09-13T01:19:34Z DEBUG cargo::core::compiler::fingerprint] filesystem up-to-date "/nix/store/bhyli42jac57wn36n7sl24n14m3ryynz-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/ansi_term-0.12.1"

Then i get a bunch of logs like
[2023-09-13T01:19:34Z INFO  cargo::core::compiler::fingerprint] fingerprint error for nickel-lang-core v0.1.0 (/build/source/core)/Build/TargetInner { benched: false, ..: lib_target("nickel-lang-core", ["lib"], "/build/source/core/src/lib.rs", Edition2021) }
[2023-09-13T01:19:34Z INFO  cargo::core::compiler::fingerprint]     err: failed to read `/build/source/target/release/.fingerprint/nickel-lang-core-c6da6289b4897f33/lib-nickel-lang-core`

    Caused by:
        No such file or directory (os error 2)

Then a bunch of normal-looking logs e.g.
[2023-09-13T01:19:34Z DEBUG cargo::core::compiler::fingerprint] fingerprint at: /build/source/target/release/.fingerprint/itertools-52f884d0b248894f/lib-itertools

And only after a ton of logs does it actually start compiling the dependencies. I'm not seeing any of the "fingerprint mismatch" logs and nothing of note immediately before the dependencies get re-compiled
The most suspicious looking thing is the "no such file or directory" errors

Since I know that release works fine, and other profiles are affected I ran a binary search changing options to dev build to ones that release use and with:
[profile.dev]
debug = false

the problem disappears.

TODO
Try a fork of crane that does not messes with timestamps etc. in the compressed files. Logic: what else could it be? It has to be it. And really, other then some purity, there's no point. Everything is tared anyway, so Nix can't see these timestamps, and any derivation running later will be necessity have time in the future. cargo doesn't care if these files were built 1 minute or 1 month in the past, as long as they're in the past, right?

Found it!!!
https://github.com/rust-lang/cargo/issues/12457
Fix/workaround: https://github.com/fedimint/fedimint/pull/3212/files
@ipetkov I don't know if there's anything crane could do about it. Probably not. Maybe it's worth pointing out somewhere. Or at least being aware of it.

The workaround means you cannot have debug info anymore? That is quite of a bummer, as I use crane also for building binaries for integration tests.

The workaround means you cannot have debug info anymore? That is quite of a bummer, as I use crane also for building binaries for integration tests.

The workaround is profile.dev.build-override, which means it affects only building build.rs script. The chance you'll really need debug symbols in build script is virtually 0. Everything else stays unaffected.

I see, thanks!

@Radvendii You should give it a try. I'm curious if it fixes it for you as well.
Since you were getting rebuilds in release mode, you either have a different problem, or you were using debug = true or something in the release build profile.

I don't know if there's anything crane could do about it. Probably not. Maybe it's worth pointing out somewhere. Or at least it's good to be aware of it.

@dpc I'm very glad you found the issue! I've opened https://github.com/ipetkov/crane/pull/388 to document the workaround in case anyone else runs into it!


Method signature in baseline changes when adding/removing a doc comment

Expected Behavior

Given I have a method that exceeds the allowed method length or method complexity
And I have suppressed the warning via my baseline.xml
When I add a doc comment (/** ... */) to the method
Then detekt should continue to pass

Observed Behavior
After adding the doc comment, the suppression no longer works. Re-running detektBaseline shows that the signature has actually changed.
Steps to Reproduce
See https://github.com/roomscape/detekt-baseline-issue, or:

Create a long method in a class
Run detekt and observe that it fails
Run detektBaseline
Run detekt and observe that it passes
Add a doc comment to the long method
Run detekt and observe that it fails
Run detektBaseline and observe that the baseline.xml file has changed

Suppression for a method without a doc comment:
    <ID>LongMethod:Example.kt$Example$fun checkFoo()</ID>

Suppression for the same method after a doc comment is added:
    <ID>LongMethod:Example.kt$Example$ fun checkFoo()</ID>

Context

I tried to add a doc comment to an existing method in a large project
I had to regenerate the detekt baseline in order to push my changes

Your Environment

Version of detekt used: 1.20.0
Version of Gradle used (if applicable): 7.4
Operating System and version: macOS 12.2.1

Yeah that's a bug. I think we tried to address it once with #4335 but that had to be reverted as it was causing problems.

It's unreproducible on the latest main.
Seems to be fixed.

Closing then, thanks!