InfluxDB stuck in full compaction process

System info: InfluxDB 1.7.0, CentOS 7
Steps to reproduce:
I upgraded from 1.6.4 a few days ago, I also needed to drop some measurements (I dropped them while running 1.6.4 but some leftovers remained). After the upgrade to 1.7.0 everything seemed fine, then I tried to drop one such measurement which would not go away (seems like #9694). After that operation (which hanged) influxdb seem to be stuck in a continuous full compaction process on the latest shard. Even after several restarts all shards had a successful full compaction (pretty fast, in a couple of minutes) but the process never finishes on the current shard.
Expected behavior:
It should eventually stop compaction which does not seem to progress.
Actual behavior:
Compaction seem to run indefinitely.
Is there a way to make influxdb stop hugging the CPU while doing this without loosing data? I will keep in mind to never delete any data since it seems to be causing problems. I hope that this will go away once the shard expires (it seem to have some problems although influx_inspect verify reports no issues), but it would be nice if influxdb would not consume CPU for months until this happens.
Besides uselessly consuming CPU it's more annoying that restarts are much slower in this state.

Could it be that full compaction fails to finish because the shard is currently being written to?
658 telegraf autogen          658         2018-11-05T00:00:00Z 2018-11-12T00:00:00Z 2019-02-10T00:00:00Z

One more question would rolling back to 1.6.4 work? I couldn't find any information related to rollbacks.

One more question would rolling back to 1.6.4 work? I couldn't find any information related to rollbacks.

I tried a rollback to 1.6.4 (it worked) but the compaction process was still running indefinitely so I'll just wait for the shard to expire and hopefully the new shard will not exhibit this problem.

Hi @conet,
Would it be possible to provide some profiles whilst your system it exhibiting this behaviour? Ideally with it running 1.7, but if you have downgraded backt to 1.6.4 and the problem still exists then 1.6.4 profiles could be helpful too.
curl -o profiles.tar.gz http://localhost:8086/debug/pprof/all?cpu=true

The output of curl -o vars.txt "http://localhost:8086/debug/vars"  would be useful too, though consider that it contains the names of things like databases and retention policies, so you can email that to me directly if you prefer (my email is edd at the company or database dot com).

@e-dard first of all, thanks for taking the time to have a look at this issue, since the shard exhibiting this behaviour just ended it's lifetime (it's duration was 7 days which was reached today) I have new information about what happened after the shard stopped receiving writes.

As you can see in the above image after 4 hours (the default setting for compact-full-write-cold-duration) there was a very short second full compaction operation which shrank the storage with about 100MB. That process finished very fast but the other full compaction operation is still running and there is no way to stop it.
I already tried exporting all the data from one instance and restoring it in another one without any success (the full compaction process started in the new instance on the same shard with the same behaviour).
I'll send you the diagnostic files via email. Shard 77 is part of the telegraf database and it's the only one having this problem. I might have some extra leads about what could be causing this: I just upgraded telegraf from 1.7.4 to 1.8.3 the same week as I upgraded influxdb. I also use the mysql plugin which changed the types of some columns and I recall to seeing something similar to #10052.
I started moving out different kinds of measurements into different databases so in the long this will make it easier to simply drop such a faulty database, if drop measurement does not work. Until now we collected everything into telegraf (it's the only collector that we use).

@conet thanks for the update. Can you run this on shard 77? Pick the most recent TSM file possible. It will have the largest -000000XYZ.tsm number.
influx_inspect dumptsm -index path/to/shard/77/0000000xy-00000000z.tsm

You don't need to provide the output here, but could you check if the max times in the first page of the output make sense, or if they're smaller than the min times, or stuck at 1970 or similar? There was a bug we recently fixed where the max times in the TSM indexes could get written incorrectly.

Yes, they make sense, I didn't see max times smaller than min times and all times seem valid from 2018-11-..., should I check the entire output for this?

@conet if it was still that bug then I think they would all be bad. Possible it might be worth having a cursory glance at all the TSM files in that shard, but I wouldn't look into it further than that. Thanks for your email, I'm looking at your attachements right now.

All files look OK, based on the files the largest one is having difficulties finishing the compaction:
du -hs /var/lib/influxdb/data/telegraf/autogen/77/*
229M	/var/lib/influxdb/data/telegraf/autogen/77/000000807-000000004.tsm
215M	/var/lib/influxdb/data/telegraf/autogen/77/000000937-000000004.tsm
231M	/var/lib/influxdb/data/telegraf/autogen/77/000001066-000000004.tsm
220M	/var/lib/influxdb/data/telegraf/autogen/77/000001195-000000004.tsm
1.9G	/var/lib/influxdb/data/telegraf/autogen/77/000001751-000000002.tsm
157M	/var/lib/influxdb/data/telegraf/autogen/77/000001751-000000003.tsm.tmp
56K	/var/lib/influxdb/data/telegraf/autogen/77/fields.idx

@conet your listing looks strange to me. Since this shard has gone cold, I would assume that the full compaction running on that shard is the final "cold" one. I would expect the resulting file to have a level higher than 3, which is what seems to be indicated by the temp file in that directory.
In fact, given the temp file in the directory I would expect a level 2 compaction to be running, but your /debug/vars output provided out-of-band indicates a full compaction is running on that shard. 🤔
If you restart your node the temp file would be cleaned up—does the full compaction startup again on that shard with the same level file? I understand you may not be able to restart your node, so don't worry if that's the case.

In fact, given the temp file in the directory I would expect a level 2 compaction to be running, but your /debug/vars output provided out-of-band indicates a full compaction is running on that shard. 🤔

It also seems strange to me I can send you the entire compaction history so you can see there were other level compactions running (you will see them in the logs).

If you restart your node the temp file would be cleaned up—does the full compaction startup again on that shard with the same level file?

Yes, in fact I just restarted the server before to be able to send debug stats, I had pprof-enabled = false. Don't worry I'm working on the copy instance (as I said I setup a copy using backup/restore hoping to solve this), as you can see after I setup the copy everything was fine, there was no unusual CPU usage but after about an hour the full compaction process started and has been running ever since.

Will send you the logs over email.

Thanks @conet; I believe this is the same underlying issue as reported in #10468. I'm going to keep this issue open though, as you're running your copy instance on 1.7.0 still, which could provide us with the ability to do some more investigation. If you could keep the instance around that would be great.

Looking at the CPU usage I noticed that the load has increased, I now have a new and different kind (level 2) of compaction process that seem to go on without stopping. Please note that this is a completely new shard written by 1.7.0.

A service restart does not make any difference:
du -hs /var/lib/influxdb/data/logs/autogen/90/*
133M	/var/lib/influxdb/data/logs/autogen/90/000000032-000000003.tsm
34M	/var/lib/influxdb/data/logs/autogen/90/000000040-000000002.tsm
33M	/var/lib/influxdb/data/logs/autogen/90/000000048-000000002.tsm
34M	/var/lib/influxdb/data/logs/autogen/90/000000056-000000002.tsm
34M	/var/lib/influxdb/data/logs/autogen/90/000000064-000000002.tsm
32M	/var/lib/influxdb/data/logs/autogen/90/000000064-000000003.tsm.tmp
33M	/var/lib/influxdb/data/logs/autogen/90/000000073-000000002.tsm
4.1M	/var/lib/influxdb/data/logs/autogen/90/000000074-000000001.tsm
4.2M	/var/lib/influxdb/data/logs/autogen/90/000000075-000000001.tsm
4.2M	/var/lib/influxdb/data/logs/autogen/90/000000077-000000001.tsm
4.0K	/var/lib/influxdb/data/logs/autogen/90/fields.idx

@conet please see my comment https://github.com/influxdata/influxdb/issues/10468#issuecomment-438658124
I'm going to close this issue, and we will use the other one for any further discussion. And thank you once again for your help with data to debug with.


RDWeb webclient (Basic) authentication

Use case: I'm evaluating options for implementing RDWeb webclient behind a loadbalancer that handles authentication and achieve the passing of credentials to the RDWeb webclient. On "Default Website\RDWeb\Pages" I can do this using "Basic Authentication" simply by changing the IIS Authentication.
Issue: It seems that "Default Website\RDWeb\webclient" allows to use "Windows Authentication" by changing "C:\Windows\Web\RDWeb\Pages\web.config", but "Basic Authentication" is not supported. (or documented?) I was unable to make Basic Authentication work.
Could basic auth be made to work?

We hugely appreciate your contribution, feedback, and support to our technical documentation.
We’ve phased out GitHub Issues on this repository as the feedback mechanism for content and have replaced it with a new feedback system. For more information see, Content User Feedback. As part of this rollout, we are migrating your issue where our team will use your comments to improve the documentation experience. Unfortunately, this does mean you won't be able to access this issue after the migration is complete.
All issues will be migrated by Thursday 29th August 2024.
Once again, thank you for your valuable feedback, we hope you’ll continue to contribute using our new feedback system.


Fix/polygon gas

Proposed changes
Adding a gas price strategy for Polygon
Fixes
n/a
Types of changes
What types of changes does your code introduce to agents-aea?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist
Put an x in the boxes that apply.

[x] I have read the CONTRIBUTING doc
[x] I am making a pull request against the develop branch (left side). Also you should start your branch off our develop.
[x] Lint and unit tests pass locally with my changes and CI passes too
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have checked that code coverage does not decrease.
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged and published in downstream modules

I guess this is the fastest way to implement this, but it might be better to create a separate ledger plugin for polygon instead. Maybe open an issue for the latter? Let's wait for others' opinions as well

Yes, I also doubted about how this should be done, especially because I was not sure if L2 chains deserve their own plugin or not. But yes, as you say it was the fastest way in order to our deployment. Let's wait for more opinions.


[FEAT] - Add sonarcloud security check

Describe the Problem
more security checks
Describe the Solution
SonarSource/sonarcloud-github-action
Optional Additional Details
No response

Doesn't support shell:

Java, JavaScript, TypeScript, Python, C#, C, C++, and a few others


⚠ Samsung has degraded performance

In 042533e, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2546 ms

Resolved: Samsung performance has improved in 1206e16.


Malwarebytes and windows defender scan it as a virus

So i just want to know from the creator if i should be worried or is it just a false positive.
<3

although im not the creator i can tell u its safe

The code is public, you can see if it is safe or not

The code is public, you can see if it is safe or not
if i knew what im looking at i would


Dynamic SSR/PWA

Describe the problem
I have a SvelteKit app, working great.
I want to be able to export the app to Capacitor, so need static assets and code.
Would be the most awesome kick-ass thing ever if I could have two build settings.

SSR normal site
App assets

And would be super-mega-awesome if this could switch automatically and the code would magically work.
Describe the proposed solution
Enable a switch to allow SSR apps to create static assets for PWAs without changing code.
Alternatives considered
:*-(
Flutter..
Importance
would make my life easier
Additional Information

Get the advantages of SSR on web
Get the instant response on PWA.

Making this feature work will bring us into a new dimension for development.
No response

I recently built a Capacitor app + PWA from the same code base, you can read a blog post about the architecture here.
I did not use SSR because it was not necessary, but there is nothing to prevent you from that. There is already a switch, set:
export const ssr = true / false

...in your src/routes/+layout.ts to enable/disable SSR. You can for example make a "pre" script before building your app for PWA/Capacitor that sets this value to either true or false. But there's a lot more complexity to it than that to have a working Capacitor/PWA, as you will see in the blog post.

@khromov thanks for the reply
Yes indeed we have made cap svelte PWAs apps before.
But we want SSR for web/performance/seo reasons while maintaining the same code-base for PWA, so the best of both worlds, otherwise using Svetekit is kinda pointless.

@khromov I noticed you're not using platform native transitions, you can do this easily in Capacitor, any reason why?

The issue with the pre-script is, how do the +server scripts work?

+server.ts files are stripped from the output if you compile with adapter-node, unless you set them as export const prerendered = true, I touch on this in the blog post. So your static export won't have any of the server code.

you're not using platform native transitions

I'm not familiar with this, is it a plugin or something in Capacitor core? You can send a link and I'll check it out.

@khromov capacitor includes native page transitions with a custom view controller: https://capacitorjs.com/docs/ios/viewcontroller. It doesn't work on simulator only real device.
This is also coming to Android (Android is only just now implementing this 15 years later)
Yes I saw that in your post export const prerendered = true but you didn't explain how that works.
Is the code then exported an executed a layer above in static JS for example?

@khromov you can use this code to make it feel more 'native' https://github.com/ionic-team/capacitor/issues/1097#issuecomment-1077649377

@ollyde If you set export const prerendered = true for a +server.ts file, it will be executed during build time both for adapter-static and adapter-node. If you set false (default) it will be executed during runtime on adapter-node and not exist at all in adapter-static export.

@khromov yes but what do you mean by executed. Do you mean server.ts will be run on the client before the page is displayed allowing data to be passed to it? If that is the case how do we show loading indicators on the client then?

@ollyde Server endpoints are never executed on the client. In the context of +server.ts together with setting export const prerender = true, if you were to build the app it would run the server endpoint at build time, on the computer where you executed npm run build, any output from this load function would be generated as a static file in the build folder.
For example if you had /src/routes/posts.json/+server.ts and you built your page to the build/ folder, you would end up with a file called build/posts.json. This data would be static and not updated after the build. I use this approach to generate my apple-app-site-association file for example, because I want to use TypeScript for the format. :-)

@khromov oh that's a shame, hence why this ticket exists.
Would be nice to have +server.ts converted to some client-side caller with loading indicator options. Of course not all +server files can do this, but const prerender = true can take care of that.
My quest is to get SSR working on hosted platforms and PWA working for apps without any configuration :D

It would be helpful if you specified how you would imagine it to work, for example, what files would be created, how would the code look (provide a "pseudo-code" example), and what would the output be like + a pipeline of how the data low.
As for your last message, "client-side caller with loading indicator options" is exactly what a +page.ts load function would do. It runs on the server and in the client and it can be used to fetch data both in adapter-static and other dynamic adapters. In dynamic adapters it would generate server side rendered HTML, and in adapter-static it would depend a bit how your build environment was set up, but the data would usually be refreshed client side. (Unless it's data that relies on for example cookies or localStorage values, such as data for logged in users)

@khromov the +page::load function only runs on the client to make requests no? Unless I missing something out here.

@ollyde +page load is called "universal" and also runs on the server, please see:
https://kit.svelte.dev/docs/load#universal-vs-server

So the PWA can also use page load, but then how do indicators and the like work?
For example if I use an infinite double way list?

"how do indicators and the like work"

You have to build your own loading indicator, you can do this for example using beforeNavigate and afterNavigate hooks.
If you prefer to do the load state in your component, you can use streaming promises:
https://kit.svelte.dev/docs/load#streaming-with-promises
All of this works with adapter-static.
You might like the official Svelte Discord channel, it's better suited for open-ended discussions imho:
https://discord.com/invite/svelte

@khromov thanks for this info. Going to give this  a try.

It seems like I am describing the same issue in https://github.com/sveltejs/kit/issues/10332

https://github.com/sveltejs/kit/issues/10332


I only get test ads even when I set isTesting to false

I need help, when I switched from admob-simple i had no problems receiving ads but now that i am using admob free, when i show my ads they always return as test ads.my app has been on the market for months and this is the first time i had this issue of only test ads showing. anyone know how to set this to false in order to display real ads?
document.addEventListener('deviceready', function(){
function adInit(){
var admobid = {}
if (/(android)/i.test(navigator.userAgent)) {
admobid = {
interstitial:'ca-app-pub-xxxxxxxx/xxxxxxxx',
rewardvideo:'ca-app-pub-xxxxxxxx/xxxxxxxxx'
};
//interstitial
admob.interstitial.config({
id: admobid.interstitial,
isTesting: false,
autoShow: false,
});

//reward video
admob.rewardvideo.config({
id: admobid.rewardvideo,
isTesting: false,
autoShow: false,})

}
adInit()
try{
admob.rewardvideo.prepare()
admob.interstitial.prepare()
}
catch(ad){}
}})

Have mediations been set up? From what I saw in the plugin's documentation here, you need to set up mediations for ads to show. At least that's what I did to receive the reward videos.

+1 same issue

any uppdate on this? i also dont see any ads in producion mode, but i do see the placeholder when isTesting is true...

Yeah, I can't get LIVE ads to show in production mode either.    Been trying for 2 weeks.

"Test Ads" work fine in the android emulator.
LIVE ads work normal when previewing in the ionic DevApp - using "ionic serve -c"
But once the app is built and installed (both signed and unsigned) on either the emulator or a device, I get nothing.

But I can't get any of the Ionic Admob plugins to work properly.
I would appreciate any suggestions as well.  Thanks.
ionic 4.5.0
cordova 8.1.2

I gave up months ago and went with admob pro. Nearly same implementation
On Dec 11, 2018 7:32 PM, "Rob Goodwin"<EMAIL_ADDRESS>wrote:
Yeah, I can't get LIVE ads to show in production mode either. Been trying
for 2 weeks.

"Test Ads" work fine in the android emulator.
LIVE ads work normal when previewing in the ionic DevApp - using
"ionic serve -c"
But once the app is built and installed (both signed and unsigned) on
either the emulator or a device, I get nothing.

But I can't get any of the Ionic Admob plugins to work properly.
I would appreciate any suggestions as well. Thanks.
ionic 4.5.0
cordova 8.1.2
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ratson/cordova-plugin-admob-free/issues/309#issuecomment-446416198,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AdF93D2pMlY3cKQFQFmyiU33vLw2cxSiks5u4E6XgaJpZM4Xswjq
.

i get some error  Please help me ASAP
HomePage.html:65 ERROR TypeError: Object(...) is not a function
at AdMobFreeBanner.config (index.js:91)
at HomePage.webpackJsonp.199.HomePage.showBannerAd (home.ts:44)
at Object.eval [as handleEvent] (HomePage.html:65)
at handleEvent (core.js:13589)
at callWithDebugContext (core.js:15098)
at Object.debugHandleEvent [as handleEvent] (core.js:14685)
at dispatchEvent (core.js:10004)
at core.js:10629
at HTMLElement. (platform-browser.js:2628)
at t.invokeTask (polyfills.js:3)

You can’t see real ads directly. It’s normal that are not be showed.
Since February 2019, google added a new rule to avoid developer from using signed apk’s on virtual machines under vpn that were clicking to earn directly with virtual clients.
So just publish your app and wait about 2 or 3 days for the review, then real ads will appear.


module system: Improve error messages around faulty imports

Description of changes

Make explicit the module "class", which identifies the module system application, such as "nixos", "darwin" or "homeManager". This allows us to print an appropriate error when importing for example a home-manager module into NixOS:
The module foo.nix#darwinModules.default was imported into nixos instead of home-manager.
I do not expect users or module authors to set the class attribute in their modules (although they may). Instead I envision a library like flake-parts to set this for us.

~Use the new class information to automatically import the appropriate default module from a flake.~

Detect other mistakes that involve the importing _type-carrying values. For instance importing a nixosSystem result into NixOps network appears to be a somewhat common mistake, and for good reason. These changes detect this scenario and explain what to do instead.

Depends on https://github.com/NixOS/nix/issues/7186
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

This is non-compositional

I'm not sure I follow. You may be referring to either commit.

If you refer to the introduction and checking of class: The purpose of nominal typing is to prevent bad compositions without having to both the user with irrelevant details.
If you refer to the treatment of flake attrsets in the module system: I've already proposed an alternative that did not couple with flakes, but that was shot down because allowing arbitrary manipulation of all modules would allow the possibility of behaviors that make modules harder to understand.

bloats the evalModules arguments

I disagree. It is optional and it will only be set in contexts where imports are expected to be used. For example, NixOS sets it only once per system, counting specialisations as systems of course. Submodules that provide the various foo.<name>.{x,y,z} don't need to set it.
class does have some use as a special arg, as it allows conditionals in modules that are otherwise identical between, say, nixos and darwin.

there is a much simpler solution #196584 (comment)

That's unnecessary logic. I'd rather read some extra attribute paths than some custom map call that doesn't transfer between classes.

I was blurring things together, let me separate them.
Re class, I think this is a good idea, but it doesn't really make sense to pass it in specialArgs. It's not (just) an argument passed to each module, it's a change to how modules are interpreted, so it should be a top-level argument to evalModules (with null default), and if it is set it can be passed to each module as well if desired.
My real objection, and the lack of modularity, is in hard-coding the m.${classModules}.default lookup. This is setting opinionated policy rather than providing composable mechanism, ironically enough a policy that you just disagreed with in one particular instance, and becomes a piece of magic that cannot be overridden nor inferred from the usage site. I still think the right way to handle this is with a map in the end-user's invocation, but it certainly doesn't need to be in a core agnostic component like the module system; e.g. we could have the map I suggested inside flake-parts.lib.evalFlakeModule

Re class, I think this is a good idea

Cool. I don't really like the name I came up with. I've also considered kind, but I feel that's too tied to value : (type : kind), although we don't have that here. sort seemed ugly but I'd be easily convinced. Maybe there's a better word?

but it doesn't really make sense to pass it in specialArgs.

It is conveniently low overhead when it comes to most evalModules calls (submodules), but I have not made a measurement.

this is setting opinionated policy

That is exactly the intent. Some things shouldn't be left to opinion.
This seems to be where we disagree.

rather than providing composable mechanism

We already have this, and a map based solution can still be used as desired. As such it does not enforce a policy, but merely encourages it.

Final question
If we have

nix run = nix run apps.default
nix devshell = nix devshell devShells.default

then wouldn't it make sense to have

imports = [ flake ]; = imports = [ flake.nixosModules.default ];

If you agree, and the module system can't know about flake attributes, we'd need a generic mechanism. I this case the disagreement would be with infinisil.
If you disagree, that choice is a small loss for users and a small win for the ivory tower. We'd have to agree to disagree, and I'll leave this pr open, so that others can form an opinion.

It is conveniently low overhead when it comes to most evalModules calls (submodules), but I have not made a measurement.

What overhead do you mean? Syntactically, { specialArgs.class = ... } seems basically equivalent overhead to { class = ... }. In terms of runtime, either we allocate an extra attr on the top-level arg to evalModules or we evaluate an extra attr on specialArgs. What's the difference?

That is exactly the intent. Some things shouldn't be left to opinion.

We shouldn't layer flake-specific knowledge into the module system.

that choice is a small loss for users and a small win for the ivory tower

I understand your disagreement with my stance, but I object to this characterization. The abstract principles I'm applying here, of composable interfaces without layering violations, are principles I hold because they result in better software and I'm applying them because I think this case is within the context that those principles are sound. The "ivory tower" comment suggests that there is some necessary conflict between theory and practice and I'm choosing theory here, but there is no such conflict.
In this particular case, I only think this is a "loss for users" in a short term sense. The ultimate cost of this decision, and more importantly making decisions like this down the road, is that the code becomes less maintainable, more tightly coupled, more brittle, less usable in new contexts. Especially since the exact same functionality can be moved to components that are actually flake-specific, I see no reason to add this here that wouldn't also result in more and more cruft over time.

It is conveniently low overhead when it comes to most evalModules calls (submodules), but I have not made a measurement.

What overhead do you mean? Syntactically, { specialArgs.class = ... } seems basically equivalent overhead to { class = ... }. In terms of runtime, either we allocate an extra attr on the top-level arg to evalModules or we evaluate an extra attr on specialArgs. What's the difference?

evalModules would have to handle the argument on every invocation and create an entry for it in an Env, not just when it is set, but in each submodule.

That is exactly the intent. Some things shouldn't be left to opinion.

We shouldn't layer flake-specific knowledge into the module system.

... and lib shouldn't have knowledge of systems, derivations, packages shouldn't know about their NixOS tests.
Yet, these violations of the implicit architectural guidelines are very helpful every day. Are those bothering you?

that choice is a small loss for users and a small win for the ivory tower

I understand your disagreement with my stance, but I object to this characterization. The abstract principles I'm applying here, of composable interfaces without layering violations, are principles I hold because I think they result in better software and I'm applying them because I think this case is within the context that those principles are sound. The "ivory tower" comment suggests that there is some necessary conflict between theory and practice and I'm choosing theory here, but there is no such conflict.

We disagree.

In this particular case, I only think this is a "loss for users" in a short term sense. The ultimate cost of this decision, and more importantly making decisions like this down the road,

Each decision should be judged by its merit.

is that the code becomes less maintainable, more tightly coupled, more brittle, less usable in new contexts. Especially since the exact same functionality can be moved to components that are actually flake-specific

It can not. It's supposed to be a convenience, not an eyesore. We've already discussed this.

, I see no reason to add this here that wouldn't also result in more and more cruft over time.

I find this disrespectful. Infinisil and I have been maintaining and improving the module system for the last couple of years, with diligence and care. Why wouldn't we continue to do so?

evalModules would have to handle the argument on every invocation and create an entry for it in an Env, not just when it is set, but in each submodule.

Ah, right. Would be nice to see measurements here but makes sense.

... and lib shouldn't have knowledge of systems, derivations, packages shouldn't know about their NixOS tests.
Yet, these violations of the implicit architectural guidelines are very helpful every day. Are those bothering you?

Nix is a language for package management. Why shouldn't lib have knowledge of systems and derivations?
As for the package issue, this does actually bother me a bit since it encourages testing only on NixOS and discourages building a testing interface that is decoupled from the distro. But there is also an important difference between a layering violation in leaf applications of a system and layering violations in a core library component, one which is actually used in many different contexts.

Each decision should be judged by its merit.

I agree, my point is that a) on this particular case, this is a small step of additional tight coupling and magic surprise and b) the reasoning applied in this case, if applied in future decisions, would equally justify further tight coupling and magic surprise.

I find this disrespectful. Infinisil and I have been maintaining and improving the module system for the last couple of years, with diligence and care. Why wouldn't we continue to do so?

I wasn't making a comment on your diligence or care, and I appreciate the ongoing work on the module system. I was making a claim about the logical implication of the reasoning being applied here, not about the quality of your efforts. If you disagree, OK, you disagree, I think that you're wrong and that the overall quality of the work will suffer as a result but that doesn't mean I think you're not overall doing an important job well.

I've pivoted the PR to type checking instead of type matching, and it now covers an extra set of errors where we can be quite helpful with our error messages.
No more layer violation.

And in addition to the points below, this should also have documentation.

It's documented to the current standard for module system documentation, which is barely above zero: code comments only.
I'll make an attempt to improve this, but I'm concerned that it will further delay this PR.

There doesn’t seem to have been any bikesheding on the class name, so here are a few suggestions (with emoji to react with to vote):

❤ domain (my favorite at the moment)
👍 variety
👀 flavor
🎉 style
(😀 if you want to keep class)
(😕 if you have other suggestions)
(👎 if you don't think we should vote now)

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2023-04-13-documentation-team-meeting-notes-41/27264/1

It's documented to the current standard for module system documentation, which is barely above zero: code comments only.
I'll make an attempt to improve this, but I'm concerned that it will further delay this PR.

Yeah that's true. In this case I think we shouldn't bother with documentation in this PR. It's weird to have documentation on the module system but only just enough to a explain a very specific advanced feature that is only just introduced.

It's documented to the current standard for module system documentation, which is barely above zero: code comments only.
I'll make an attempt to improve this, but I'm concerned that it will further delay this PR.

Yeah that's true. In this case I think we shouldn't bother with documentation in this PR. It's weird to have documentation on the module system but only just enough to a explain a very specific advanced feature that is only just introduced.

Do you think it actually ended up weird? I think lib.evalModules is a reasonable initial scope for the docs. I could un-include the file if you think it isn't ready for viewing by users, but to me that feels counterproductive. Something is better than nothing.

Do you think it actually ended up weird?

Hmm yeah somewhat, because the covered concepts are not very relevant to most module system users. But in the docs team we also agreed to really just have reference docs in the NixOS/Nixpkgs/Nix manuals, so it actually fits into that model. Tutorials for new users should be in another place then.
So yes, I take that back, let's have these docs :)

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/tweag-nix-dev-update-47/27387/1

Anything left to do here?
(Except documenting the rest of the module system, which is a project of its own)

I'd like to do another review, but I don't always have the time for that. I'll try to get to it within the next week though, ping me if I forget

Thanks for the many reviews :heart:

Here's a PR for the postponed types.configuration

https://github.com/NixOS/nixpkgs/pull/230377

External use of lib.modules.pushDownProperties is deprecated. If your use case isn't covered by non-deprecated functions, we'd like to know more and perhaps support your use case well, instead of providing access to these low level functions.

@roberth as far as i can tell, my use case isn't covered by non-deprecated functions. as you expected, external uses of these are a bit gnarly, but i'll try here to condense my justification for its use.
config = mkMerge <list> at module scope often causes infinite recursion, e.g. when the items in <list> are computed attrsets: a well-known issue but with AFAIK no widely agreed upon solution. code like this barfs:
{ config, ... }:
{
  options.my.option = mkOption {
    type = types.listOf types.string;
  };

  config = mkMerge (map generateMoreConfig config.my.option);
  # ^ ERROR: `config.*` depends on `config.my.option` => infinite recursion
}

assume generateMoreConfig produces an attrset with assertions and users. the simple fix is:
  config = let
    genAssertions = conf: (generateMoreConfig conf).assertions;
    genUsers = conf: (generateMoreConfig conf).users;
  in {
    assertions = mkMerge (map genAssertions config.my.option);
    users = mkMerge (map genUsers config.my.option);
  };

however, this fails if generateMoreConfig includes any lib.mkIf at the toplevel, e.g.
generateMoreConfig = value: lib.mkIf value != "root" {
  assertions = ...; users = ...;
};

hence, any more robust equivalent to module-scope mkMerge has to "push down" that outer lib.mkIf:
generateMoreConfig = value: {
  assertions = lib.mkIf value != "root" ...;
  users = lib.mkIf value != "root" ...;
};

but you don't want to manually push these down. ideally one could take the original, clean-but-unsupported code:
config = mkMerge (map generateMoreConfig config.my.option);

and defeat the infinite recursion by constraining the attrnames at the outer layer -- without contorting the logic everywhere below that:
config = let
  mergeEachAttr (map generateMoreConfig config.my.option);
in {
  assertions = mergeEachAttr.assertions;
  users = mergeEachAttr.users;
};

hence, this mergeEachAttr function has to use lib.pushDownProperties. in case anyone wants to dive deeper, my concrete use:

helper definition: mkTypedMerge
where it's used

@uninsane That seems like a valid use case. Could you create an issue for it?

@uninsane That seems like a valid use case. Could you create an issue for it?

the mkMerge limitations are i think well-known (but with no foreseeable direct "fix"). unless i'm not thinking in the right scope, the only new/actionable issue i can see is "pushDownProperties was marked private even though a single external user still depends on it". but i don't know that that's even an issue: i only felt compelled to say anything about it because you signaled in the code change that you'd like to hear from anyone who's using these. i assumed the code committer would then be in a position to make any relevant judgements.
if you prefer i formalize that feedback through some issue, i'm happy to do that, but i'd need guidance because i have so much less context here than you that i don't know what the substance of any issue here actually is.

My goal is just to have a separate thread for it, not to put you through the bureaucratic wringer or something. This thread already has over a hundred messages, and changing the topic of a long thread makes things harder to find. GitHub hiding the middle also doesn't help.

limitations are i think well-known (but with no foreseeable direct "fix")

While there's an element of there being a fundamental problem that prevents the simplest fix imaginable, that doesn't stop is from finding practical solutions, like you did!

but i'd need guidance because i have so much less context

Most issues are conversations. They don't have to be full-on research presentations. You might fill in the template and paste your existing prose in the appropriate header(s). Markdown source is under the comment edit button ;)


Open add component line edit by double-click

Resolves #1908.
Change the quick add component feature so that it requires double-click instead of single-click. This makes it less likely to appear by accident.

Fixed by forwarding event to parent class.


Changing to AdaptiveAvgPool2d on SqueezeNet and ResNet

Hi,
I'm dealing with this problem with the hardcoded AvgPool2d on SqueezeNet, some people notice this a year ago, also in ResNet, so I made these changes, Pytorch must be as robust as possible. I left the other AvgPool2d layers because they have another purpose.
nn.AvgPool2d -> nn.AdaptiveAvgPool2d((1,1)) assures the output's size like (batch_size, num_classes, 1,1) on SqueezeNet and ResNet.

@houseroad will this change break ONNX?
I'm inclined to make all torchvision models take an nn.AdaptiveAvgPool2d(1, 1) instead of a fixed AvgPool2d with fixed kernels.

Thanks to you and your work!!


go test -cover outputs build errors with incorrect line numbers

What version of Go are you using (go version)?
go version go1.8 linux/amd64

Does this issue reproduce with the latest release?
Yes, I tried the same commands with go1.9 and the exact same output resulted.
go version go1.9 linux/amd64

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/leighmcculloch/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build083996345=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?
$ git clone https://github.com/leighmcculloch/go-strrev.git
$ git checkout broken # ba1a9a9962908c303407878762304821fd764c4a
$ go test -cover

What did you expect to see?
I expected to see the same build error output as when I run go test.
$ go test
# 4d63.com/strrev
./strrev.go:17: undefined: bytes
./strrev.go:19: undefined: s
FAIL    4d63.com/strrev [build failed]

What did you see instead?
Different build error output, which displayed different line numbers and different locations for the files.
$ go test -cover
# 4d63.com/strrev
/tmp/go-build636495490/4d63.com/strrev/_test/_obj_test/strrev.go:16: undefined: bytes
/tmp/go-build636495490/4d63.com/strrev/_test/_obj_test/strrev.go:19: undefined: s
FAIL    4d63.com/strrev [build failed]

Closing as dup of #6329.


ImageProcessor images lose quality

Hi James, I've some problem about the quality.
The quality of the resized picture is not the best, I tryied before with an image jpeg
http://mb-website.com/prova.html
and then with one .jpg took from my iphone X.
http://mb-website.com/prova2.html
Could I add something to get a better result?
Thank you
ImageProcessor (2.8.0), ImageProcessor.Web (4.11.0), ImageProcessor.Web.Config (2.6.0), I dont use visual or whatever program, everything done by myself, the web.config is:
<?xml version="1.0" encoding="UTF-8"?>

<configuration>

  <configSections>
 <sectionGroup name="imageProcessor">

    </sectionGroup>
  </configSections>

  <system.web>
    <httpModules>
      <add name="ImageProcessorModule" type="ImageProcessor.Web.HttpModules.ImageProcessingModule, ImageProcessor.Web"/>
    </httpModules>
  </system.web>

  <system.webServer>
  <validation validateIntegratedModeConfiguration="false" />
    <modules>
    <add name="ImageProcessorModule" type="ImageProcessor.Web.HttpModules.ImageProcessingModule, ImageProcessor.Web" />
    </modules>
  </system.webServer>

<imageProcessor>
    <security configSource="config\imageprocessor\security.config" />
    <caching configSource="config\imageprocessor\cache.config" />
    <processing configSource="config\imageprocessor\processing.config" />
  </imageProcessor>

</configuration>

The cache.config, and processing.config are not modified.
In the security.config I have added
      <whitelist>
             <add url="http://images.pexels.com"/>
      </whitelist>

I put them in root/config/imageprocessor

@mirko2609
A couple of points I must make before we engage in any future conversation.

Don't change issues on-the-fly. You opened an issue with one problem and have now changed it to another doing so is a waste of maintainer time.
Don't ever email me to pester me to look at your issues. You opened your first issue at 12:35am local time Sydney and emailed me only 2 hours later asking me to look at it. You can tell where in the world I live and the time zone I am subject to  because it is listed on the same page as my email address (I have since removed that email address). You have since sent me subsequent emails. That is not acceptable behavior.

Now, in answer to your issue. You've reduced the image dimensions by 33% so have removed a large portion of the pixel data. The System.Drawing resampling method does not work particularly well when resizing certain images but we are stuck with it so cannot change anything there. I would suggest enabling and using the GaussianSharpen web processor to sharpen up the images slightly after resizing.

Thank you for the answer and sorry for my behaviour, I dont use to stay on this forum so that s why I tryied to push up my question, I wasnt sure you still check this forum, sorry again.
I have tried with sharpen=10&sigma=1.5&threshold=10 or sharpen=100&sigma=1.5&threshold=10 but looks they are not better then before.
Yes, the picture with coronza it has be resized a lot but the other not, they dont have both professional quality

No worries.
I've opened a new issue #792 to cover this as the new rewrite produces better output. I'll investigate the cause of the difference using your images.


Setting min/max property can change current month, but doesn't emit event

If you set the min or max properties of <calendar-date>, this can have the effect of automatically changing the currently showing month, because the current or default month is no longer accessible.
The problem with that is if you're rendering your own <select> elements in the header for changing the year and month, you need to know when the month/year changes, to be able to update the selected options in those elements.
As far as I can tell though, no event is emitted in this scenario, which would be ideal. Similar to "focusday".
The way I'm trying to get around this is by having a function which I call to set the max/min property, and at the same time update my <select> elements. The problem is I don't actually know what month is definitely showing, because by default the focusedDate property is undefined, and doesn't change as a result of setting the max or min properties.
Any guidance would be appreciated!

Can you provide a codepen or similar showing your issue? I don't have a great deal of spare time to dig into this myself at the moment, and a minimal reproduction would really help. Thanks!

Yes, no problem - here's a codepen:
https://codepen.io/gavrichards/pen/QWXPRag
It logs into a box when the focusday or change events occur, or when you click the "Log .focusedDate" button.

Click "Set Min to 2024-12-10"
Note that the calendar changes to December, but nothing is logged
Tab around and see that the 10th receives focus
Click the "Clear Min" button
Note that the calendar changes back to September, but still nothing is logged
Click the "Log .focusedDate" button and note that the value is remains undefined

You can also navigate to October and see the focusday event is emitted, but still setting the min and clearing it doesn't emit anything, despite changing the calendar to December and then back to October.
I hope this is useful, and thank you for your time.

I've been digging into the code, specifically CalendarDate and useCalendarBase.
What I think I've learnt is that the current showing month could be deduced by using in order of availability: focusedDate, then value, then today, and then "clamping" that to between min and max.
In which case I can probably replicate that in my own code which wraps the calendar component.
It would be great if the calendar emitted a focusedDate (focusday event) which took into account the min and max though.
Thank you!

Thanks! I'll take a look at this later in the week and try to find a reasonable solution.
I'm not overly keen on firing events, since they are generally reserved for user initiated changes not programmatic changes. But it may be the only option here. Will keep you posted!


ChainerMN's ImageNet example is broken because of update in pickle module

Conditions

Chainer >= 5.0.0

Code to reproduce
The ImageNet example on Python >= 3.6

import multiprocessing
p = multiprocessing.Process(target=lambda *x: x, args=())
p.start()

Error messages, stack traces, or logs

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kfukuda/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/home/kfukuda/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/kfukuda/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/multiprocessing/context.py", line 291, in _Popen
    return Popen(process_obj)
  File "/home/kfukuda/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
  File "/home/kfukuda/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/kfukuda/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/multiprocessing/popen_forkserver.py", line 47, in _launch
    reduction.dump(process_obj, buf)
  File "/home/kfukuda/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x7fb633f35e18>: attribute lookup <lambda> on __main__ failed

For details, see https://docs.python.org/3.6/library/pickle.html
The following types can be pickled:

* None, True, and False
* integers, floating point numbers, complex numbers
* strings, bytes, bytearrays
* tuples, lists, sets, and dictionaries containing only picklable objects
* functions defined at the top level of a module (using def, not lambda)
* built-in functions defined at the top level of a module
* classes that are defined at the top level of a module
* instances of such classes whose __dict__ or the result of calling __getstate__() is picklable (see section Pickling Class Instances for details).

Thus lambda is not pickable.

I failed to reproduce. No error was raised from the code below.
Environment:
chainer==5.0.0
cupy-cuda92==5.0.0.
Python 3.6.6 |Anaconda, Inc.| (default, Oct  9 2018, 12:34:16)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> p = multiprocessing.Process(target=lambda *x: x, args=())
>>> p.start()
>>>

Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> p = multiprocessing.Process(target=lambda *x: x, args=())
>>> p.start()
>>>

I can not reproduce with just :

import multiprocessing
p = multiprocessing.Process(target=lambda *x: x, args=())
p.start()

But I have to say I also have the same problem, please run the training script if you want to reproduce the error. And hope someone could found a solution


Fails with companion tickets

When you have a companion ticket linked to the ticket, you aren't able to change the flight (southwest posts a message that says a companion ticket is linked and you must cancel that ticket first).  This causes the script to fail on line 69 of southwest.py:
INFO:root:Attempt 1 at capturing headers....
[I:pyppeteer.launcher] Browser listening on: ws://<IP_ADDRESS>[snipped]
[I:pyppeteer.launcher] terminate chrome process...
INFO:root:Detected account XXXXX and captured headers.
[I:pyppeteer.launcher] terminate chrome process...
.INFO:root:Processing: XXXXXX
.Traceback (most recent call last):
  File "app.py", line 285, in <module>
    check_for_price_drops(user.username, user.password, user.email, user.headers, user.cookies, user.account)
  File "app.py", line 143, in check_for_price_drops
    cancellation_details = southwest.get_cancellation_details(record_locator, passenger['first-name'], passenger['last-name'])
  File "/path/to/script/southwest-alerts/southwestalerts/southwest.py", line 69, in get_cancellation_details
    token=temp['viewReservationViewPage']['_links']['cancelBound']['query']['passenger-search-token']
TypeError: 'NoneType' object is not subscriptable

It would be nice if the script continued on to the next confirmation code instead of dying as I have other reservations in the future that I'd like to check.

Here's what I did to fix the problem above, let me know if you want a PR (see bold):
        temp = self._session.get(url)
        **# check if there is a greyboxmessage warning for companion or other issue
        if not (temp['viewReservationViewPage']['greyBoxMessage'] is None):
            return None**
        url = '/api/mobile-air-booking/v1/mobile-air-booking/page/flights/cancel-bound/{record_locator}?passenger-search-token={token}'.format(
            record_locator=record_locator,
            token=temp['viewReservationViewPage']['_links']['cancelBound']['query']['passenger-search-token']
        )

And the corresponding continue to the for loop in app.py (see bold)

            # try:
            cancellation_details = southwest.get_cancellation_details(record_locator, passenger['first-name'], passenger['last-name'])
            **if cancellation_details is None:
                continue**

I don't currently have a companion so haven't been able to test, but would love if you could try to push this fix!  Thanks!

I'll have to cherry pick cause I've got a lot of differences in my code (using Twilio for SMS notifications and config.ini instead of system variables for my southwest login info) so it may take me a bit, but I'll create a pull request here soon!

I don't think this is related directly to the companion pass.   I get the same error for a non-companion pass flight.
Hoopsbwc34, i'd love to see your changes and integration for Twilio thou!