Question: Low volume  with this preset?

Thanks a lot for your effort and the promising preset.
I justed switched my LENOVO T14s Gen. 1 under MANJARO to PipeWire and installed EasyEffects with some Preset-packages, as the original sound under MANJARO is terrible. Strange for me: Running WINDOWS under MANAJRO and VirtualBox, the same .mp3-File sounds OK.
Now I tried with EasyEffects and could improve the quality with different presets:
$ easyeffects --presets
Ausgabe-Voreinstellungen:
Advanced Auto Gain,
Bass Boosted,
Bass Enhancing + Perfect EQ,
Boosted,
MakeSoundGreat,
Loudness+Autogain,
Perfect EQ,
PipeWire EasyEffects Default,
laptop_unsuck,
Eingabe-Voreinstellungen:

My Problem: The volume running laptop_unsuck is much lower in comparison to the other presets. Typically I have 30% volume but I need about 90% to achieve a similar level with laptop_unsuck. How can I solve this? Do you have a recommendation, how to decide between the presets or is there a tutorial for further improvement?
Thank you!

Go adjust the sliders and numbers in the preset stages if some of the frequencies are too quiet for you.
I don't know much about audio but I do understand that for an equalizer, you raise and lower a multiplier (volume) for various frequencies and somehow interpolate those adjustment points with a wave function.
While this is not an equalizer, it does do something similar (it is a superset of an equalizer). For the audio to sound "balanced", certain frequencies had to be lowered (made quieter) to make others more prominent. The quietness is expected.

Thanks a lot for your reply:

Go adjust the sliders and numbers in the preset stages if some of the frequencies are too quiet for you.
But my issue is different:

PipeWire is installed, EasyEffects is running, the volume for my system is set to 30%, I'm playing a MP3-File.
I switch to preset Advanced Auto Gain: I can hear the sound
I switch to preset Bass Boosted: I can hear the sound a bit more bass boosted, but almost with similar volume.
I switch to preset laptop_unsuck: I hear nothing at all, only after setting the speakers volume to 90% I can hear it, but very quiet!

How to solve this?

It's possible that the T14s Gen1 has different speakers than the Gen2. In this case it's likely that the preset in this repository will make the sound worse instead of improving it.
I'm not an audio expert either, but as @spikespaz already said, a decrease in overall volume is expected if you want to boost the lower frequencies (but not as strong as you describe).
If you want to get the best result for the speakers of your model, I recommend to follow the guide mentioned in the readme (in EasyEffects hit F1 for help and select Enhancing your internal notebook speakers without using an Equalizer).

I switch to preset laptop_unsuck

Maybe you are using the old version from the gist instead of the current version from this repository? It looks like it since the preset file is now named thinkpad_unsuck.
This might explain the very low volume, since the parameters for the filter plugin changed in EasyEffects 7.08: https://github.com/wwmm/easyeffects/blob/master/CHANGELOG.md#708
Please make sure you are using the latest preset version, the filter mode should be RLC (BT):
https://github.com/sebastian-de/easyeffects-thinkpad-unsuck/blob/6356f4953b54911e156f31ed90da29eb4436ad7e/thinkpad-unsuck.json#L21

Great, you made it:

Maybe you are using the old version

I had this one:
laptop_unsuck.json
Using the following, it's OK:
laptop_unsuckNEU.json
Thanks a lot!

Great, you're welcome!

What is this NEU file?
https://github.com/sebastian-de/easyeffects-thinkpad-unsuck/files/13770555/laptop_unsuckNEU.json

It's this one works fine.
The URL for the one, that doesn't work for me can be found here. AS JSON does typically not allow comments, may be the revision-ID could be part of the filename in future?

AS JSON does typically not allow comments, may be the revision-ID could be part of the filename in future?

Well, one of the advantages of git is that you don't have to do these kind of things.
I manage all my dotfiles using git (this repository just contains a copy of a single preset I use).
But you can just use curl to get the latest version:
curl 'https://raw.githubusercontent.com/sebastian-de/easyeffects-thinkpad-unsuck/main/thinkpad-unsuck.json' > ~/.config/easyeffects/output/thinkpad_unsuck.json


Development test frontend threebot deployer

Related issue https://github.com/threefoldtech/js-sdk/issues/1701

Codecov Report

Merging #1794 (d99de2d) into development (1104c02) will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           development    #1794   +/-   ##
============================================
  Coverage        32.35%   32.35%
============================================
  Files              134      134
  Lines            12855    12855
============================================
  Hits              4159     4159
  Misses            8696     8696

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1104c02...d99de2d. Read the comment docs.


Fix ped bug

This also fixes the bug where the tag size is being displayed incorrectly when an undo command is executed after add/delete tag command.

LGTM


Implement GitHub webhooks

This PR allows Towtruck to listen for several types of events on the POST /api/github/webhooks endpoint. This can be used to incrementally update the database and spreading out the requests it makes to the GitHub API, rather than issuing a large number of requests in a short window of time as it does when seeding.
The events it listens for are:

pull_request.opened and pull_request.closed to update the list of open pull requests
issues.opened and issues.closed to update the list of open issues
push and issues.edited to update the list of dependencies
dependabot_alert to update the list of vulnerabilities
repository to update basic information about repositories

NB: this pull request depends on #63, which should be merged first.

Thanks for tackling this feature, Dan, there's a lot of complexity at work and it looks like your refactoring and test improvements will improve readability and maintainability of the codebase. I've found it a little tricky to review due to the number of changes here (+1,484 −67!).

Suggestion: Would it be possible to split the big refactor into its own PR made up of smaller commits?
Here’s how I’d structure it:

Add deleteAllRepositories function
Add deleteAllDependencies function
Use new functions in seed script.

I’d find it much easier to digest the changes in these smaller chunks, and could hopefully give you a more thorough review.
This PR could then consist of a few smaller commits, one for each of the Github events.


Issue with FSDP in Evaluation Mode for Mixtral7*8B Model

Hello,
I am encountering an issue when attempting to use Fully Sharded Data Parallel (FSDP) with a 56-billion-parameter sparse MoE model in evaluation mode. The model is too large to fit on a single GPU, and when I enable FSDP in evaluation mode, it does not seem to function properly.
Upon setting evaluation_mode to False, I found that FSDP works correctly. This behavior is related to the following code snippet from the accelerate library:
https://github.com/huggingface/accelerate/blob/31fd2b1ad6b9c1cd1480568399a311b3caaf62dc/src/accelerate/accelerator.py#L1341

evaluation_mode (bool, optional, defaults to False):
Whether or not to set the model for evaluation only, by just applying mixed precision and
torch.compile (if configured in the Accelerator object).

Based on the description, it seems that setting evaluation_mode to False should not affect precision during the evaluation task. Can you confirm if my understanding is correct? Moreover, is there a known issue with FSDP in evaluation mode, or could you suggest any workaround to use FSDP with such a large model in evaluation mode?
Thank you for your assistance.

@muellerzr


deal with circular dependency

Some shared libraries have circular dependencies that cause an infinite loop in the current implementation of magicpak.
I added a HashSet that memorize which library is being resolved or resolved.
The following is a case with circular dependency that I encouraged.
# ldd /usr/lib/libfreetype.so.6
	linux-vdso.so.1 (0x00007ffd15bd8000)
	libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f00c8c78000)
	libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f00c8c41000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f00c8c27000)
	libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007f00c8b53000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f00c898a000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f00c8844000)
	libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007f00c881d000)
	libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f00c86e9000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007f00c8d9d000)
	libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007f00c8677000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f00c8655000)
# ldd /usr/lib/libharfbuzz.so
	linux-vdso.so.1 (0x00007ffe5a1db000)
	libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007fd7d18e2000)
	libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007fd7d18bd000)
	libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007fd7d1789000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007fd7d15c0000)
	libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007fd7d15ad000)
	libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007fd7d1576000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007fd7d155a000)
	libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007fd7d14e8000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fd7d14c6000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007fd7d1ac8000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007fd7d1380000)

Sorry for the late reply. Thanks!

Sorry for the late reply. Thanks!


Ability to add "regions" or "areas" to a tilemap

Is your feature request related to a problem? Please describe.
Similar to walls, it would be useful to paint zones on a tilemap that could then be accessed by event handlers (on sprite enter area, on sprite exit area, etc). This could be for detecting when a character has walked "into" a town, for example.
Describe alternatives you've considered
We're currently using the "cover all tiles with [image]" block (in the tilemaps extension) to obscure tiles that form a "region". This is a little clunky, and requires new blocks for each type of cover tile generated.

Here's my pitch for why we should keep the "cover all tiles with [image]" block:
Let's imagine you're making a zelda-like game. You have a few locations in your game: a dungeon, an overworld, and a shop. When you're adding these locations to your game, you need to have tiles that act as "doors" between the locations. When the player overlaps your "door" tiles, you want to load the next location. This is a very common game scenario.
With the "painting regions" approach, I pick a color for each door going in each direction. For example:

Red = door from overworld to dungeon
Blue = door from dungeon to overworld
Green = door from overworld to shop
Orange = door from shop to overworld

Now when I put this in my code, I select those colors and code the door behavior between each of these maps. I need to remember each color and what it's supposed to do whenever i look at my code. If I forget, I need to go hunting between my three tilemaps to see what each color means. Also, I might run out of these regions pretty quick if I start adding other features to my game.
On the other hand, with the "cover all tiles" approach, I can just make tiles like this:


Remove deprecated geo search features

https://github.com/elastic/elasticsearch/pull/22876

Closed via #2613


Debugging Unit Tests Raises Exceptions

When running unit tests without debugging, the tests pass.  When debugging unit tests, an exception is raised in what appears from external testing to be valid code.
Running or debugging the same tests in PyCharm or from the command line does not demonstrate this erroneous behavior.
Downgrading the Python extension from version 2021.10.x to 2021.09.124x fixes the issue.
VSCode is version 1.61.0
Unittest Framework
sys.version = '3.8.5 (default, Sep  4 2020, 02:22:02) \n[Clang 10.0.0 ]'

I should add:  I'm using the unittest framework.

@wahono77 Can you provide repro steps?

If you run into this issue, please try toggling the "User Uncaught exceptions" setting. Also, if this continues to not do what you expect, please file a bug in https://github.com/microsoft/debugpy repo.


Possible wrong field value in simple.cmd template?

Nick P brought up in conversation the fact that in all of the condor jobs, the executable was set to "simple.sh", whereas in the old jobsub_client/server model, the condor_q output showed the user executable.
In this line:
https://github.com/marcmengel/jobsub_lite/blob/65d47a6dc74445b68f60f8ddf394ac649a05d164/templates/simple/simple.cmd#LL5C24-L5C35
We can see that the executable field is set to script_name, and defaults to simple.sh.  Doing a git grep on the repository, though:
$ git grep script_name
lib/dagnabbit.py:                    thesevalues["script_name"] = f"{name}.sh"
templates/simple/simple.cmd:executable         = {{script_name|default('simple.sh')}}

We just get those entries.  Are we using the wrong variable here, or should we be setting script_name somewhere?

script_name gets set in dagnabbit.py when we render multiple wrapper files; but executable= has to be set to the wrapper, even in old jobsub.   So if the Cmd attribute is getting set to something other than the wrapper, old jobsub  must be doing it some other way...

I'm going to change this from a "bug" to an enhancement, since for the jobsub users, this is behaving OK.  We can look into changing how it looks in the condor output, but it's not a priority now.


fabric api 0.87.0 keeps crashing

Describe the bug
Game keeps crashing on launch, it finishes loading Forge and then it shuts down.
Steps to reproduce

Install Forge 47.1.43
Download Forgified Fabric API latest version and place in mods folder
Try launching the game
...

Logs
No crash log, only ''exit code 1''
Additional context
No response

Works for me, it properly shows the loading error screen, along with the version range error. The Forgified Fabric API requires Forge 47.1.3 specifically.


Help with use of WIKIDATAFACTS

Hello! with the WIKIDATAFACTS function, is it possible to grab the taxon common name in Indonesian, shown underlined here in red?
Help with the syntax would be very welcome for both :

opt_multiObj | string= | Whether to return all object values (pass all) or just the first (pass first) when there are more than one object values (optional).
opt_properties | Array opt_properties Limit the resulting facts to a list of properties (optional).

I don't understand the instructions, and there isn't an example in the docs that uses the string= argument.

It is indeed  possible.
=WIKIDATAFACTS(P3,"first","P1843")
where P3 is id:Cipoh_kacat
However, the help documentation for certain functions such as this, is too short.
It will be great if the documentation can be in more detail and descriptive. :)

Thanks for helping out, @viswam. I have a backlog of open issues that I want to address at some point, so any help with improving the docs is welcome. Here is the to-be-expanded example section. You need to check the file locally to make sure the indentation with the <pre> works fine. I'm closing this very Issue.


Github Verification

When i try and link my Github accout with Keybase for the Xlm airdrop. I get the message "cant find proof please try again" i have followed the instructions and created a public gist can someone please processs or help me with this issue.
link below of my public gist.
https://gist.github.com/faz123786/b125127c77b020ce5f90202aa1e04101

I’m in the same situation, I need help too


Add a null check to method TestUtils.compressHeapDumps.

Motivation:
java.io.File.listFiles() may return null and cause a unexpected NPE.
Modification:
Add a null check for variable files. And if variable files is null, the compressHeapDumps method will just return after logging a error message.
Result:
Fix the potential NPE.

@netty-bot test this please


fix: Show error when signing in with unregistered email

What changed?

Added a check to verify if the email exists before attempting the sign-in process.
Display an error message if the email is not registered, prompting users to register a new account.
If the email is not found, users are provided with a link to the registration tab for account creation.
Fixes #296

Why?

This change was implemented to improve the user experience by notifying users immediately when they attempt to sign in with an unregistered email.
Users no longer need to guess why they can't log in or be confused by no feedback, leading to better usability and fewer failed attempts.

Screencast:
https://github.com/user-attachments/assets/0354c7b0-7a4e-47c6-a60a-b697a66a3e88

Note you'll also need to add a changeset to this PR! You can do it from the command line with pnpm changeset.

The output :-
https://github.com/user-attachments/assets/0354c7b0-7a4e-47c6-a60a-b697a66a3e88

This commit will close this issue https://github.com/namesakefyi/namesake/pull/299/commits/fa90cdf0242bf2cfbaadc04602dbc3c534c7fc4b

Thanks so much @nakul-py! 💟


Feature: use environment variables to set arguments values

First step to #213

Codecov Report

Merging #215 (5ed6978) into main (ca8934e) will decrease coverage by 5.09%.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #215      +/-   ##
==========================================
- Coverage   68.92%   63.83%   -5.09%
==========================================
  Files          33       33
  Lines        1670     1684      +14
  Branches      350      352       +2
==========================================
- Hits         1151     1075      -76
- Misses        431      525      +94
+ Partials       88       84       -4

Impacted Files
Coverage Δ

qgis_deployment_toolbelt/commands/deployment.py
84.78% <100.00%> (+0.33%)
:arrow_up:

qgis_deployment_toolbelt/commands/upgrade.py
63.88% <100.00%> (-14.99%)
:arrow_down:

qgis_deployment_toolbelt/utils/str2bool.py
100.00% <100.00%> (ø)

qgis_deployment_toolbelt/utils/win32utils.py
18.84% <0.00%> (-49.28%)
:arrow_down:

...loyment_toolbelt/jobs/job_environment_variables.py
66.66% <0.00%> (-20.52%)
:arrow_down:

qgis_deployment_toolbelt/profiles/shortcuts.py
50.00% <0.00%> (-18.75%)
:arrow_down:

qgis_deployment_toolbelt/utils/file_downloader.py
60.00% <0.00%> (-2.50%)
:arrow_down:


[WIP] Canonical url solver architecture

Refined version.
Please see full description and discussion at https://github.com/scrapinghub/crawl-frontier/pull/23

@shaneaevans I really need your input here ;)

Idea looks good to me, I can see we'll need a more sophisticated version of this for our larger crawls. Needs tests.

Great! What do you think about the name: canonical solver? Is it ok from english prospective?

Name is ok from the English perspective.  I prefer a name like CannonicalURLMapper, which makes it clear that this may map URLs to other ones, not only perform transformations (e.g. like Scrapy https://github.com/scrapy/scrapy/blob/master/scrapy/utils/url.py#L38 )


TypeError: Cannot read property 'ImageUrl' of undefined

@rdegges
Hello, when I updated coinmarket cap v1 to v2 on line 54 in the app.js file, it threw up this message
"TypeError: Cannot read property 'ImageUrl' of undefined"
On looking further, the coinmarketcap data has been changed from an array to an object.

Want to submit a PR with the changes? I'd be happy to mege!


Client reconnection on server going away (1001)

Hi! I'm trying to use your implemetation of auto-reconnection in client:
Here is my code
async for websocket in connect(
    server_url,
    extra_headers={"Authorization": 'some_code'}
):
    try:
        await websocket.send('get_some_data')
        async for msg in websocket:
            print(msg)
    except ConnectionClosed:
        logic_to_stop()

When i shutdown server i get code 1001 and client starts connecting again:
< CLOSE 1001 (going away) [2 bytes]
= connection is CLOSING
> CLOSE 1001 (going away) [2 bytes]
= connection is CLOSED
! connect failed; reconnecting in 3.0 seconds

However i thought that client should throw ConnectionClosed (or maybe any other exceptions) exception and i can stop client from reconnecting. Is there a way to use reconnection with some stopping mechanism?

Nvm, i used finally block with close_code check


How to handle julian->gregorian calendar switch?

Driver version
<IP_ADDRESS>
Redshift version
PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.34355
Client Operating System
Mac OS Monterey 12.0.1
JAVA/JVM version
Java 11.0.12
Table schema
c1 date
Problem description
I would like to get missing dates during julian->gregorian calendar switch without casting to string in Redshift side.
JDBC trace logs
n/a
Reproduction code
Tried getString, getDate, getObject(LocalDate) and getTimestamp, but all of them returned 1582-10-15 (not 1582-10-05).
I know TO_CHAR function returns the correct result as string, but I don't want to use it.
        Connection connection = DriverManager.getConnection(JDBC_URL, USER, PASSWORD);
        Statement statement = connection.createStatement();

        statement.execute("DROP TABLE IF EXISTS est_date");
        statement.execute("CREATE TABLE test_date (c1 date)");
        statement.execute("INSERT INTO test_date VALUES ('1582-10-05')");
        ResultSet rs = statement.executeQuery("SELECT * FROM test_date");
        while (rs.next()) {
            // returns 1582-10-15
        }

Thanks for reporting issue. We will look into it.

As column is DATE type, application treat value as java.sql.Date. Per definition this class represents: A milliseconds value represents the number of milliseconds that have passed since January 1, 1970.
https://docs.oracle.com/javase/8/docs/api/java/sql/Date.html
So looks like TO_CHAR is the way to go or if you have home grown application use resultset.getString() method.

@iggarish I would recommend providing the right value via ResultSet.getObject(, Class<T> type).

@iggarish Please reopen this issue.

what's the Class type can be use here?

@iggarish MySQL and PostgreSQL JDBC return correct values via LocalDate.

This is a known issue in the switch from the Julian to Gregorian calendars in Java as explained here: https://stackoverflow.com/questions/23975205/why-does-converting-java-dates-before-1582-to-localdate-with-instant-give-a-diff and here: https://stackoverflow.com/questions/23457470/calendar-to-date-conversion-for-dates-before-15-oct-1582-gregorian-to-julian-ca
As a workaround, you can perform a cast on your query when fetching such as:
SELECT CAST(c1 AS TIMESTAMP) FROM test_date
And then get the right date via:
ResultSet.getObject(, LocalDate.class)

@bhvkshah I knew the workaround, see the description. I would recommend checking other JDBC drivers' implementation more.

@ebyhr PG JDBC has this implementation which already exists in Redshift JDBC Driver: https://github.com/pgjdbc/pgjdbc/issues/1534
Could you point me towards a specific driver's code which has the fix you would like to see implemented?

@bhvkshah
Please take a look at  https://github.com/pgjdbc/pgjdbc/blob/96824b0693e7dcf209fb16f909c3cabfb125c240/CHANGELOG.md#4234-2022-04-01-141628--0400

Thanks for the additional information, we will take a look

Hi @ebyhr , we will not be pursuing this issue in the near future, and have decided to close it for now. It will be reopened if we come back to this issue. The Redshift Driver team appreciates your contribution and thanks you for bringing this to our attention!


[dependencies] Axios Cross-Site Request Forgery Vulnerability

Dependabot has reported a new moderate vulnerability stemming from axios. This vulnerability is not automatically resolvable, and manually overriding the version of axios to the fixed version below causes an error when running "test-storybook"

Hey there @seanmc86! Thank you for reporting.
That's quite weird, as axios is a dependency of wait-on, which is only used as dev dependency of the test-runner, not direct dependency. Regardless, I updated wait-on to fix this issue for the test-runner codebase here, and it will be included in a patch release soon, maybe it helps?

Thank you for the quick update!
I am still seeing the issue after installing "@storybook/test-runner": "^0.14.2-next.0" – looks like it is because of jest-playwright-preset and it's subsequent dependencies (jest-playwright-preset -> jest-process-manager -> wait-on -> axios`
axios  0.8.1 - 1.5.1
Severity: moderate
Axios Cross-Site Request Forgery Vulnerability - https://github.com/advisories/GHSA-wf5p-g6vw-rhxx
No fix available
node_modules/jest-process-manager/node_modules/axios
  wait-on  5.0.0-rc.0 - 7.1.0
  Depends on vulnerable versions of axios
  node_modules/jest-process-manager/node_modules/wait-on
    jest-process-manager  *
    Depends on vulnerable versions of wait-on
    node_modules/jest-process-manager
      jest-playwright-preset  >=1.0.0-rc1
      Depends on vulnerable versions of jest-process-manager
      node_modules/jest-playwright-preset
        @storybook/test-runner  *
        Depends on vulnerable versions of jest-playwright-preset
        node_modules/@storybook/test-runner

@prstn good note on jest-playwright-preset which was last published a year ago, and seems unmaintained. In fact, they recommend migrating to @playwright/test


feat: fire a validated event on validation and add shouldSetInvalid method

Description
1.Fires a validated event when the web component's validation takes place.
2.Adds a protected shouldSetInvalid() method. It will be used for preventing the web component from modifying the client-side invalid state on the Flow side afterward.
Fixes # (issue)
Type of change

[ ] Feature

Checklist

[ ] I have read the contribution guide: https://vaadin.com/docs/latest/guide/contributing/overview/
[ ] I have added a description following the guideline.
[ ] The issue is created in the corresponding repository and I have referenced it.
[ ] I have added tests to ensure my change is effective and works as intended.
[ ] New and existing tests are passing locally with my change.
[ ] I have performed self-review and corrected misspellings.

Additional for Feature type of change

[ ] Enhancement / new feature was discussed in a corresponding GitHub issue and Acceptance Criteria were created.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: roastedcpu:x: mukherjeesudebiYou have signed the CLA already but the status is still pending? Let us recheck it.


Proc not available on macOS – consider psutil

I arrived at this repo via Stack Overflow. While I'm aware it's a good few years old, I figure I'm not the only one who'll land here.
Currently, process-watcher gets its list of pids from /proc. Given that /proc is unavailable on macOS, I'd suggest psutil as as good cross-platform alternative.
Example of its usefulness: psutil.pid_exists(12345)
I've been looking to extend process-watcher with support for Pushover so I may do this myself and submit a pull request, if I find time.
Thanks for sharing.

@jrmedd Thanks, I haven't worked on this project in a long time, if you're interested in maintaining it I could transfer it to you.


Parallelizing imports

I've really enjoyed using your project, thanks for sharing it! Definitely the most responsive photo gallery I've used so far 🙂
I noticed that the import process is currently single threaded. Is there any reason why it couldn't be parallelized across multiple threads? There's a worker_threads API that looks interesting, I haven't used it before: https://nodejs.org/docs/latest-v14.x/api/worker_threads.html

Thank you for your positive feedback. Glad that you like the responsiveness, too.
Could you describe in detail where you experience slow processing and what is your expectation? What exactly would you propose to parallelize?

I'm using the Docker Compose setup on my server, so to import more pictures I run:
docker compose run gallery run import --update

I'm currently at the stage where I'm adding hundreds of pictures and videos from various backups, so the import takes several hours. I forgot to take logs so I don't have an exact time, sorry about that :sweat_smile:. I ran top during the import and I noticed that only one of my CPU cores were being used. I'm assuming the import process is limited by the speed of your CPU, so I was wondering if we could process multiple pictures at once using multiple CPU cores. I know that node.js is single threaded, however it looks like the worker_threads offers a way to create multiple threads so you can make use of more CPU cores.
Is there anything in the import process that relies on the images being processed in a specific order? Or could it be parallelized across multiple CPU cores?

You are using the import with arg --update which is not recommended to use at the beginning. So I am asking myself: do you want to optimize/fasten the import and UX process or do you want to optimize the CPU usage? The first should be fixable with the --initial arg. If you have question on that, please let me know.
Further I assume that the focus is to view the personal media archive. The process to import it is a required step but can consume time - or in other words: My assumption is that the import is not time critical.
IMHO the most time consuming tasks are video transcoding and object/face/similarity detection which should be already parallelized. I can not remember but I assume that the video trancoding with ffmpeg will consume more than one thread. Further the object/face/similarity detection depends on your setting. This could be parallelized if required by spawning and proxy multiple API nodes, true. Eg. via nginx.
For me, using worker_thread optimizes the gallery on some corners where it is not so important at the end. If you host your photos at home you are most probably willing to accept a waiting time if the result - browsing the gallery itself - is fine. You can afford to wait until the last image is resized and the last video is tanscoded and imported.
Of cause you might have a different opinion and PRs are welcome to fix your needs.
I hope is not to destructive or demotivating. Please continue the discussion if you like. And I am happy for further inputs. Thank you for reaching out and for using the web gallery.

Hi @xemle, sorry for not being clear before. I've already run the import --initial command for my initial import. Since that's done, I'm running import --update for each new batch of photos. On my last run of import --update I noticed only a CPU core was in use. I provide 4 to the VM I run the gallery in, so I was thinking it may be possible to speed up the import process.
You're definitely right, time is not critical and I am willing to wait for the processing to finish. But, I'm also always interested in how the system works and if there's any opportunity for (hopefully) small changes that improve performance.
I did not check CPU usage when the videos were being imported but I'm willing to bet you're right that ffmpeg parallelizes video transcoding by default. I believe I checked when the import process was importing images. So yeah, not asking you to change anything, more just wondering about the internals of the import process. On my next import I'll try to take better notes, maybe I'll be able to make the modifications myself :smile:

Thank you very much for the further details. I am happy that you are used --initial before and we share the basic expectations and interests.
Also the image resizing should be using multiple CPU cores using sharp package (see its fast notes). But to be honest I never checked how to speedup the import process in detail.
If you are interested in optimizing the import we could do also a 1:1 session talking about the internals and share ideas.

I appreciate the offer! I'll let you know if I think that's necessary after I've poked around more.


Fixed invalid syntax for creating a cron job returning null

When creating a cronjob with an invalid syntax, it used to just return null, so a user wouldn't know why the cron job was not created.
I've modified this so it now throws an error when there is an invalid cronjob trying to be created so the user knows what the issue is.

@toymachiner62 Not throwing errors was a deliberate design decision. You can still check the validity by ensuring that create() didn't return null and job.isValid() == true

Interesting. What is the reason for allowing invalid syntax? I'm just
curious.
On Sunday, November 8, 2015, Blagovest Dachev<EMAIL_ADDRESS>wrote:

@toymachiner62 https://github.com/toymachiner62 Not throwing errors was
a deliberate design decision. You can still check the validity by ensuring
that create() didn't return null and job.isValid() == true
—
Reply to this email directly or view it on GitHub
https://github.com/dachev/node-crontab/pull/18#issuecomment-154929143.

--
Tom Caflisch
507.202.0087
<EMAIL_ADDRESS>http://allthingswebdesign.com

create() does not allow invalid syntax, it's simply chooses to return a null instead of throwing an exception.
Also, validation capabilities should not be overestimated. There are many versions and flavors of crontab, each with its own options and supported syntax.

I suppose that's fair since we check for job === null && !job.isValid().

I haven used this module in a while but this should work :-)
require('crontab').load(function(err, crontab) {
  var jobs = crontab.jobs();

  for (var i = 0; i < jobs.length; i++) {
    crontab.remove(jobs[0]);
  }

  crontab.save(function(err, crontab) {

  });
});

Let me know if it doesn't and I'll look into making this possible.

That worked like a charm. Thanks!

No problem.


More accurate convertion

My problem is when e.g. converting from Euro to Serbian Dinar I get a pretty accurate value but if I want to convert from Serbinar Dinar to Euro which has the multiplier around 0.0085 it gets kinda inaccurate because the result given from the library is always rounded to 2 digits (in this case 0.01).
As solution I would like a more accurate result.

Hey man @hirschnico,
I've fixed your issue by adding withoutRounding boolean.
If you want full amount then you can set this flag to true.
await CurrencyConverter.convert(       from: Currency.usd,       to: myCurrency,       amount: 1,       withoutRounding : true,     )


[WIP] Add support for using Ubuntu 16 as a base instead of Debian 8

This requires building Python 3.4 from source, so we do that.
Future changes may add support for other OS base images, such as Debian 9 and Ubuntu 18.  We don't support arbitrary OS base images in general, so it's an enumerated list.

Not ready for review, despite what GitHub thinks :)

Ready for review


sf apex run test : error - "Always provide a classes, suites, tests, or testLevel property"

Summary
In CI/CD after the metadata was deployed to scratch org when running apex tests it is error saying "Error (1): Always provide a classes, suites, tests, or testLevel property."
Command we are using to run apex unit tests is:
sf apex run test -o $username -l RunLocalTests -c -r junit -w 90 -d ./tests/apex
Username is present because before running a tests I'm writing to debug a string with username variable, so it is always displaying like this: "***** APEX TESTS FOR<EMAIL_ADDRESS>STARTED *****"
Actually if restart the pipeline several times it can start running apex unit tests, but there is no concrete number how many times pipeline should be restarted
Expected result
Apex unit tests execution started
Actual result
Error (1): Always provide a classes, suites, tests, or testLevel property.
System Information
Which shell or terminal are you using? - powershell
{
  "architecture": "win32-x64",
  "cliVersion"<EMAIL_ADDRESS>  "nodeVersion": "node-v20.12.2",
  "osVersion": "Windows_NT 10.0.17763",
  "rootPath": "C:\\Windows\\System32\\config\\systemprofile\\AppData\\Local\\sf\\client\\2.41.8-dd2c629",
  "shell": "cmd.exe",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.17 (core)",
    "@oclif/plugin-commands 3.3.2 (core)",
    "@oclif/plugin-help 6.0.21 (core)",
    "@oclif/plugin-not-found 3.1.8 (core)",
    "@oclif/plugin-plugins 5.0.19 (core)",
    "@oclif/plugin-search 1.0.23 (core)",
    "@oclif/plugin-update 4.2.11 (core)",
    "@oclif/plugin-version 2.0.18 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.16 (core)",
    "@oclif/plugin-which 3.1.9 (core)",
    "@salesforce/cli 2.41.8 (core)",
    "apex 3.1.11 (core)",
    "auth 3.6.9 (core)",
    "data 3.3.4 (core)",
    "deploy-retrieve 3.6.13 (core)",
    "info 3.2.7 (core)",
    "limits 3.3.5 (core)",
    "marketplace 1.2.5 (core)",
    "org 4.1.8 (core)",
    "packaging 2.4.2 (core)",
    "schema 3.3.6 (core)",
    "settings 2.2.6 (core)",
    "sobject 1.4.1 (core)",
    "source 3.3.9 (core)",
    "telemetry 3.3.5 (core)",
    "templates [56](https://gitlab.thelem-assurances.fr/thelem/salesforce/shiva/-/jobs/55195#L56).2.5 (core)",
    "trust 3.6.8 (core)",
    "user 3.5.5 (core)"
  ]
}

Hey @OlegBogdanovych, I tried to reproduce this with the Dreamhouse sample app and was not able to.
After creating the scratch org and deploying, I ran:
sf apex run test -l RunLocalTests -c -r junit -w 90 -d ./tests/apex
(I had my --target-org set to default at scratch org creation time)
Can you please provide the output of the same command with the --dev-debug flag?

Also, if you are able to replicate the issue with Dreamhouse and provide steps, that would be very helpful. Thanks!


or() is not accepting query builder factories

I have pre-0.24.0 code that constructs a query from a series of orWhere() calls that accept query builder factories. When I try to translate them to v0.24.2, I get the following error:
Type '(qb: QB) => QB' is not assignable to type 'Expression<SqlBool>'.
How do I use query builder factories in the new compound conditionals? (and() also has the problem.)
EDIT: However, I was using query builder factories because I needed a way to nest conjunctions and disjunctions. It may be possible for me to rewrite the code to return Expression instances instead of query builder factories.

The new expressions seem like they'll greatly simplify my code, but I'm losing types. I'm wondering if the new where logic bindings have to be generated from within a query builder. Consider this:
    async function selectMany(expr: Expression<any>): Promise<User[]> {
      const sqb = db.selectFrom("users").selectAll();
      const result = await sqb.where(expr).execute();
      return result as User[];
    }

    const userIDs = await db
      .insertInto("users")
      .values(USERS)
      .returning("id")
      .execute();

    const users = await selectMany(({ or, cmpr }) =>
      or(cmpr("name", "=", USERS[0].name), cmpr("id", ">", userIDs[0].id))
    );

I get this error:
Argument of type '({ or, cmpr }: { or: any; cmpr: any; }) => any' is not assignable to parameter of type 'Expression<any>'.ts(2345)
Binding element 'or' implicitly has an 'any' type.ts(7031)

Expression<any> clearly isn't able to provide the types for or and cmpr. Any suggestions for me?

Okay, I have things working. I was able to ditch a little code and a lot of test cases. Much appreciated!
But I do have a breaking suggestion, which I'll make in another issue.


fix: crash in deactivate page router outlet

We have already discuss it with @ADjenkov and he will further investigte but for now this fix seems to be quite safe.


Documentation mistake: Model::save does not return a boolean only

Describe the bug
The documentation states that Model::save returns a boolean value. This is not true. It also may return the ID of an inserted record and therefore an integer type.
CodeIgniter 4 version
CodeIgniter4 Alpha 2
Affected module(s)
Model

Closed by #1808


Feature Request: multi-epoch alternative to tf.QueueBase.close()

Examples on the web demonstrate signaling the end of queue data by calling queue.close() in the data producer and then catching the tf.errors.OutOfRangeError exception in the data consumer.
This works fine for a single epoch, but I do multiple epochs, alternating between training data and testing data, and I can't reuse the queue after calling queue.close().
The two solutions that I have thought of using the existing code are:

enqueue() some sentinel at the end of an epoch in the data producer and then tf.Assert() against the sentinel and catch the tf.errors.InvalidArgumentError in the data consumer.
know the number of enqueue's for the epoch and only dequeue that number.
Both seem a little hacky.

Multi-epoch use of queues might be simplified by adding one of the following:

A queue.reset(), that throws one tf.errors.OutOfRangeError on dequeue() or some other exception.
A queue.close(reset=True), that only throws one tf.errors.OutOfRangeError on dequeue() or some other exception.

example usage of 1):
q = tf.FIFOQueue(...)
placeholder = ...
enqueue_op = q.enqueue(placeholder)
....

def producer(data_dir, sess, q, enqueue_op, placeholder):
  for ...:
    sess.run(enqueue_op, {placeholder:...})
  sess.run(q.reset())

def do_epoch(data_dir, learn):
  threading.Thread(target=producer, args=(data_dir, sess, q, enqueue_op, placeholder)).start()
  while True:
    try:
      sess.run(...)
    exception tf.errors.OutOfRangeError:
      break

for epoch in range(NUM_EPOCHS):
  ... = do_epoch(TRAIN_DIR, learn=True)
  ... = do_epoch(TEST_DIR, learn=False)

cc @ebrevdo @mrry @josh11b

Why not build a separate graph with its own queue for eval?  You'll also get the flexibility of being able to add additional flexibility to append the eval graph for eval purposes.

Thank you for the suggestion. I guess I stayed away from two graphs to avoid code duplication between train and eval graphs and the accompanying maintenance issues, and because it would require updating the eval model weights with the trained model weights. I could define them both within one graph and share the weights, but that feels messy and still has the duplication.
With a single model, I can just run sess.run([loss, train_op],...) for training, or leave off train_op for eval sess.run([loss],...). It seems clean; only one place to modify the model and weights are ready from the previous training invocation.

If this isn't how others are structuring their code, then there is no urgency/necessity. I have been pre-computing the number of enqueue()'s that will be performed in an epoch, and then running the consumer the appropriate number of times, with asserts that the producer is not alive and the queue is empty before starting the next epoch.

People usually use one method to create the core of the code, and a
separate method to take its output and generate the loss and gradients. A
third to generate the eval losses.  This way you can share code and
variables between the two graphs.  See tf.contrib.learn.
On May 26, 2016 3:05 PM, "Mark Woodward"<EMAIL_ADDRESS>wrote:

If this isn't how others are structuring their code, then there is no
urgency/necessity. I have been pre-computing the number of enqueue()'s that
will be performed in an epoch, and then running the consumer the
appropriate number of times, with asserts that the producer is not alive
and the queue is empty before starting the next epoch.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-222008578

I think I see what you are saying. Just to be certain, do you by chance have a specific example? I looked through tensorflow/contrib/learn/python/learn but did not find an example of this on my first scan. No worries if not, thank you.

I have exactly the issue that markpwoodward does. If you have tensorflow v0.8 you might be able to use this workaround where by default you stream directly from the train queue except sometimes you pass test data in through a feed_dict. To be fair, I haven't tried it, and I came here looking for a better solution. I imagine it would look like this:
input= tf.place_holder_with_default( training_dequeue, shape)
###Do training until end of epoch
sess.run( model.update)

###Do test/validate epoch
test_batch=sess.run( test_dequeue)
sess.run( model.accuracy, feed_dict={input: test_batch} )

Picking up this old thread, a question: why don't you use make_template and have three queues feeding into three automatically shared model graphs? True, you still need to keep track when an epoch ends and switch over to the e.g. validation queue, but this can be done rather simple by either counting or have the queues return a third element that indicates one full pass through the data (training or validation data) is done. Or is this approach considered harmful in some way?

I ended up doing the switching between validation and train in a python
producer thread, and counting the number of elements processed. But I have
been thinking about this a bit, and I am leaning towards a similar (maybe
same) idea as yours, where there are two queues (train and validation),
with a tf.cond switching between queues, controlled by a placeholder. This
would allow for a single model graph (no need for multiple instantiations
with shared weights). Funny enough, Rohit Girdhar just emailed the list
with a question, and it seems that this is the setup he is using. I copied
the relevant portion below.
"I'm training a deep network with two data input pipelines, one for
training and one for validation. They use shuffle_batch_join and
batch_join respectively for parallel data reading. The data stream that
is used in the network is decided using a tf.cond operation on top of
these two pipelines, which is controlled by a is_training placeholder
that is set to true for a training iteration and false when doing
validation. I have 4 threads for reading training data and 1 thread for
validation..."
On Mon, Jul 18, 2016 at 12:21 PM, Christian<EMAIL_ADDRESS>wrote:

Picking up this old thread, a question: why don't you use make_template
and have three queues feeding into three automatically shared model graphs?
True, you still need to keep track when an epoch ends and switch over to
the e.g. validation queue, but this can be done rather simple by either
counting or have the queues return a third element that indicates one full
pass through the data (training or validation data) is done. Or is this
approach considered harmful in some way?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-233430140,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGgTpS5Jyv_NEidwew9thKg9x5KTJAiRks5qW9IhgaJpZM4Inwby
.

Oops, tf.cond may be the wrong choice, as ebrevdo mentions in the parallel
thread which I will stop referencing now. From the documentation, it seems
that tf.cond executes both paths up to, but not including, the final
operation, so it wouldn't work as I had hoped.
Perhaps it could still work if what was passed to tf.cond() was the
queue.dequeue(), or queue.dequeue_many(), operations for the train_queue
and validation_queue. But those may themselves be the end of other tf
operations, so it might still not work.
On Mon, Jul 18, 2016 at 5:59 PM, Mark Woodward<EMAIL_ADDRESS>wrote:

I ended up doing the switching between validation and train in a python
producer thread, and counting the number of elements processed. But I have
been thinking about this a bit, and I am leaning towards a similar (maybe
same) idea as yours, where there are two queues (train and validation),
with a tf.cond switching between queues, controlled by a placeholder. This
would allow for a single model graph (no need for multiple instantiations
with shared weights). Funny enough, Rohit Girdhar just emailed the list
with a question, and it seems that this is the setup he is using. I copied
the relevant portion below.
"I'm training a deep network with two data input pipelines, one for
training and one for validation. They use shuffle_batch_join and
batch_join respectively for parallel data reading. The data stream that
is used in the network is decided using a tf.cond operation on top of
these two pipelines, which is controlled by a is_training placeholder
that is set to true for a training iteration and false when doing
validation. I have 4 threads for reading training data and 1 thread for
validation..."
On Mon, Jul 18, 2016 at 12:21 PM, Christian<EMAIL_ADDRESS>wrote:

Picking up this old thread, a question: why don't you use make_template
and have three queues feeding into three automatically shared model graphs?
True, you still need to keep track when an epoch ends and switch over to
the e.g. validation queue, but this can be done rather simple by either
counting or have the queues return a third element that indicates one full
pass through the data (training or validation data) is done. Or is this
approach considered harmful in some way?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-233430140,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGgTpS5Jyv_NEidwew9thKg9x5KTJAiRks5qW9IhgaJpZM4Inwby
.

What can work is a cond(filter_predicate, lambda: queue.enqueue(..),
tf.no_op).  This allows controlling what goes into a queue.  Most other
ways of combining queues and tf.cond usually don't have the behavior you
would expect.
On Jul 18, 2016 8:20 PM, "Mark Woodward"<EMAIL_ADDRESS>wrote:

Oops, tf.cond may be the wrong choice, as ebrevdo mentions in the parallel
thread which I will stop referencing now. From the documentation, it seems
that tf.cond executes both paths up to, but not including, the final
operation, so it wouldn't work as I had hoped.
Perhaps it could still work if what was passed to tf.cond() was the
queue.dequeue(), or queue.dequeue_many(), operations for the train_queue
and validation_queue. But those may themselves be the end of other tf
operations, so it might still not work.
On Mon, Jul 18, 2016 at 5:59 PM, Mark Woodward<EMAIL_ADDRESS>wrote:

I ended up doing the switching between validation and train in a python
producer thread, and counting the number of elements processed. But I
have
been thinking about this a bit, and I am leaning towards a similar (maybe
same) idea as yours, where there are two queues (train and validation),
with a tf.cond switching between queues, controlled by a placeholder.
This
would allow for a single model graph (no need for multiple instantiations
with shared weights). Funny enough, Rohit Girdhar just emailed the list
with a question, and it seems that this is the setup he is using. I
copied
the relevant portion below.
"I'm training a deep network with two data input pipelines, one for
training and one for validation. They use shuffle_batch_join and
batch_join respectively for parallel data reading. The data stream that
is used in the network is decided using a tf.cond operation on top of
these two pipelines, which is controlled by a is_training placeholder
that is set to true for a training iteration and false when doing
validation. I have 4 threads for reading training data and 1 thread for
validation..."
On Mon, Jul 18, 2016 at 12:21 PM, Christian<EMAIL_ADDRESS>wrote:

Picking up this old thread, a question: why don't you use make_template
and have three queues feeding into three automatically shared model
graphs?
True, you still need to keep track when an epoch ends and switch over to
the e.g. validation queue, but this can be done rather simple by either
counting or have the queues return a third element that indicates one
full
pass through the data (training or validation data) is done. Or is this
approach considered harmful in some way?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-233430140
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AGgTpS5Jyv_NEidwew9thKg9x5KTJAiRks5qW9IhgaJpZM4Inwby

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-233519829,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim7VModT2soQ10NwRw9Zf7-umSE5Rks5qXEKFgaJpZM4Inwby
.

With make_template there won't be multiple copies of the graph. Tensorboard nicely shows that the two queues (one for training data, the other for validation data) share one graph with the weights (True, you need to have two different smaller pieces of loss expressions that are associated with the queue and are placed on top of the one graph that has the trainable parameters). Deciding which queue is used is done on the outer training iteration loop, no tf.cond or any other expression-based logic necessary, you just sess.run the respective loss expression and the underlying queue is polled.

Christian, just a clarification, doesn't calling make_template twice create
duplicate operation paths in the graph, one for training and one for
validation? That duplication is probably harmless, it just seems messy when
all we want to do is switch between two input sources (although I don't
have an alternative to propose).
On Tue, Jul 19, 2016 at 12:05 AM, Christian<EMAIL_ADDRESS>wrote:

With make_template there won't be multiple copies of the graph.
Tensorboard nicely shows that the two queues (one for training data, the
other for validation data) share one graph with the weights (True, you need
to have two different smaller pieces of loss expressions that are
associated with the queue and are placed on top of the one graph that has
the trainable parameters). Deciding which queue is used is done on the
outer training iteration loop, no tf.cond or any other expression-based
logic necessary, you just sess.run the respective loss expression and the
underlying queue is polled.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-233548788,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGgTpRXoUlDAdmhT6zbDrHRd5_ReufOaks5qXHdXgaJpZM4Inwby
.

@markpwoodward hmm, i don't know what an operation path is. Say you call it twice, with two different input tensors (e.g. coming from a deque op on your training/validation set respectively). What is happening is that in the first call, the graph with the parameters is constructed. In the second call, this graph is simply reused. If you open up tensorboard, you see exactly this view, sort of a bottlenecked picture (if your shared graph is followed by some additional, loss-connected ops, which you need to have duplicated, that's true): At the bottom the two queues and their nodes, both feeding into the network expression that is made up of shared parameters, then going splitting up again to two loss-connected paths. With respect to messiness, I thought it the least messy solution, as I only had to add a for loop over the queues and kept the rest the same, make_template taking care of the rest. But messiness is probably quite subjective :-).

@osdf, thank you for the response. Would you mind include the picture from
tensorboard? I certainly may be missing something about how tensorflow
execution happens. Also, maybe brief pseudo code for your usage of
make_template(); where the goal of the pseudo code is just inference (no
loss or training), but on two different queues.
On Tue, Jul 19, 2016 at 11:16 AM, Christian<EMAIL_ADDRESS>wrote:

@markpwoodward https://github.com/markpwoodward hmm, i don't know what
an operation path is. Say you call it twice, with two different input
tensors (e.g. coming from a deque op on your training/validation set
respectively). What is happening is that in the first call, the graph with
the parameters is constructed. In the second call, this graph is simply
reused. If you open up tensorboard, you see exactly this view, sort of a
bottlenecked picture (if your shared graph is followed by some additional,
loss-connected ops, which you need to have duplicated, that's true): At the
bottom the two queues and their nodes, both feeding into the network
expression that is made up of shared parameters, then going splitting up
again to two loss-connected paths. With respect to messiness, I thought it
the least messy solution, as I only had to add a for loop over the queues
and kept the rest the same, make_template taking care of the rest. But
messiness is probably quite subjective :-).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-233719206,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGgTpRXQZW9sXwpodVkpt3Joe28JgAnrks5qXRRygaJpZM4Inwby
.

You may be able to use QueueBase.from_list to dynamically select which queue to dequeue from, see:
https://www.tensorflow.org/versions/r0.9/api_docs/python/io_ops.html#QueueBase.from_list

I don't know how I missed that! Thank you. I just tested it. It works great; two FIFOQueues, one placeholder to select the queue.
This feature request was side tracked a bit, I will leave it open as my original request of a way to signal the last dequeue of an epoch, without needing to count dequeue's, still stands. Low priority, since it is easy enough to count dequeue's and this feature is likely less relevant for larger datasets, where people don't usually do things on epoch boundaries.

@markpwoodward Can you explain how to use the feature pointed out by @josh11b? A code snippet would be great.

Hi, I am using the same solution that was mentioned here, having a boolean placeholder is_training, and combining the training and evaluation queues using batch = tf.cond(is_training, lambda: training_batch, lambda: testing_batch). Why would this not work? It seems to work for me, but I didn't check if it always dequeues both batches.

@danijar Assuming training_batch and testing_batch are the results of calls to tf.train.batch() that occur before the tf.cond(), it will always dequeue both batches. See here for an explanation (and substitute "dequeue operation" for "assignment").

Thanks. So I basically need to pass functions into tf.cond that do a tf.identity and perform the desired calls within the tf.control_dependencies of the identity, correct? What about something like this instead:
queue = tf.cond(is_training, train_queue, test_queue)
batch = queue.dequeue_many(batch_size)

The issues that I have with the two original proposals are:
The two solutions that I have thought of using the existing code are:

enqueue() some sentinel at the end of an epoch in the data producer and then tf.Assert() against the sentinel and catch the tf.errors.InvalidArgumentError in the data consumer.

What to do with batches? ie the single rows are handled by a buffering batch creator

know the number of enqueue's for the epoch and only dequeue that number.
Both seem a little hacky.

Again with batching, If the epoch size is not a multiple of the batch size, then part of the next epoch ends up on the last batch.
Perhaps an alternative solution to the OP would be a queues that raises OutOfRangeError but stays open. That way the local variable for epochs can just be reset.

One solution that I have found is to create a new input_producer each epoch and then to initialize_variables only the new local variable (the epoch count). This works but yields warnings for the old queues:
       	 [[Node: input_producer_18/fraction_of_32_full_Dequeue = QueueDequeue[_class=["loc:@input_producer_18/input_producer/fraction_of_32_full/fraction_of_32_full"], component_types=[DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](input_producer_18/input_producer/fraction_of_32_full/fraction_of_32_full)]]
W tensorflow/core/framework/op_kernel.cc:940] Out of range: FIFOQueue '_37_input_producer_18/input_producer/fraction_of_32_full/fraction_of_32_full' is closed and has insufficient elements (requested 1, current size 0)

Regarding the side track, not the original request.
@muhammadzaheer, here is an example usage of QueueBase.from_list() that you asked for.
@danijar, I would recommend using QueueBase.from_list() over tf.cond().
import tensorflow as tf

q1 = tf.FIFOQueue(capacity=100, dtypes=[tf.int32])
input1 = tf.placeholder(tf.int32, [])
enq1 = q1.enqueue(input1)

q2 = tf.FIFOQueue(capacity=100, dtypes=[tf.int32])
input2 = tf.placeholder(tf.int32, [])
enq2 = q2.enqueue(input2)

select_q = tf.placeholder(tf.int32, [])
q = tf.QueueBase.from_list(select_q, [q1, q2])
data = q.dequeue()

with tf.Session() as sess:
  # enqueue values (these would typically be in their own thread(s))
  q1_vals = [1,2,3]
  print "q1_vals = " + str(q1_vals)
  for v in q1_vals:
    sess.run(enq1, { input1: int(v) })

  q2_vals = [4,5,6]
  print "q2_vals = " + str(q2_vals)
  for v in q2_vals:
    sess.run(enq2, { input2: int(v) })

  # run an op that pulls from the queue, specifying which queue
  for batch in range(3):
    print "q1.dequeue = " + str(sess.run(data, {select_q: 0}))
    print "q2.dequeue = " + str(sess.run(data, {select_q: 1}))

outputs
q1_vals = [1, 2, 3]
q2_vals = [4, 5, 6]
q1.dequeue = 1
q2.dequeue = 4
q1.dequeue = 2
q2.dequeue = 5
q1.dequeue = 3
q2.dequeue = 6

QueueBase.from_list may be deprecated soon.  +@mrry
On Sep 9, 2016 1:14 PM, "Mark Woodward"<EMAIL_ADDRESS>wrote:

Regarding the side track, not the original request.
@muhammadzaheer https://github.com/muhammadzaheer, here is an example
usage of QueueBase.from_list() that you asked for.
@danijar https://github.com/danijar, I would recommend using
QueueBase.from_list() over tf.cond().
import tensorflow as tf
q1 = tf.FIFOQueue(capacity=100, dtypes=[tf.int32])
input1 = tf.placeholder(tf.int32, [])
enq1 = q1.enqueue(input1)
q2 = tf.FIFOQueue(capacity=100, dtypes=[tf.int32])
input2 = tf.placeholder(tf.int32, [])
enq2 = q2.enqueue(input2)
select_q = tf.placeholder(tf.int32, [])
q = tf.QueueBase.from_list(select_q, [q1, q2])
data = q.dequeue()
with tf.Session() as sess:
enqueue values (these would typically be in their own thread(s))
q1_vals = [1,2,3]
print "q1_vals = " + str(q1_vals)
for v in q1_vals:
sess.run(enq1, { input1: int(v) })
q2_vals = [4,5,6]
print "q2_vals = " + str(q2_vals)
for v in q2_vals:
sess.run(enq2, { input2: int(v) })
run an op that pulls from the queue, specifying which queue
for batch in range(3):
print "q1.dequeue = " + str(sess.run(data, {select_q: 0}))
print "q2.dequeue = " + str(sess.run(data, {select_q: 1}))
outputs
q1_vals = [1, 2, 3]
q2_vals = [4, 5, 6]
q1.dequeue = 1
q2.dequeue = 4
q1.dequeue = 2
q2.dequeue = 5
q1.dequeue = 3
q2.dequeue = 6
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-246026683,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim4OS5APEuGuTpj9-uSnIFfE0qZ4Lks5qob4vgaJpZM4Inwby
.

My current suggestion is to
group(*[cond(..., lambda: q.enqueue(....), tf.no_op) for q in queues])

See the new tf.contrib.training.bucket function for an example.

For me, @osdf 's make_template funcionality was key. @markpwoodward here is a picture of tensorboard you asked for:

Thanks @TimZaman for the image, I overread the request by @markpwoodward (apologies for that!).

Note that anyone using it must be extremely careful. Running a 'summary operation' would already pull both graphs through and dequeue both. Just fixed that issue in my implementation.

Many thanks to all contributors & to @markpwoodward , @ebrevdo, @danijar.
Since this thread is around 4 months old, curious to know the final solution you settled on- any updates on dong this more efficient/simpler?
Very grateful for any inputs.

There is now a set of methods in tf.train that allow conditional enqueue.  E.g. tf.train.maybe_batch.  the conditional may be a boolean Tensor. This is probably the appropriate solution from now on.

Many thanks @ebrevdo for the prompt response. My question was around the "side tracked" question in this thread: how do you efficiently execute training & val testing using queues in your training loop.
I was unable to get the intuition of your comments in this context (is it relevant, please)?
Currently i'm using tf.place_holder_with_default( training_dequeue, shape) method - ugly.
Details earlier in this thread link
Most thankful for any pointers.

@rsethur you can use make_template to generate two pairs of inputs and outputs, hence two independent queues
training_outputs = make_template("model", training_inputs)
testing_outputs = make_template("model", testing_inputs)

Thank you @yaroslavvb!
I implemented this and it works.
The challenge is that now model construction time is doubled! Is this expected?
This could be a deal breaker for complex models(for some reason my model construction time in GPU is more than CPU)
Any advise is appreciated.

@rsethur Using the make_template() method constrained me too much (also implemented trn and val models with different feeds. You can achieve the same thing with scoping. I.e. something like this where Model is an object setting up your model.
with tf.name_scope(const.STAGE_TRAIN) as stage_scope:
    train_model = Model(const.STAGE_TRAIN, FLAGS.croplen, nclasses, FLAGS.optimization, FLAGS.momentum)
with tf.name_scope(const.STAGE_VAL) as stage_scope:
    val_model = Model(const.STAGE_VAL, FLAGS.croplen, nclasses)

@rsethur make_template will double the number of non-variable ops in the graph. I'm guessing if you have 50k ops, you'll see an extra 20 seconds added to your graph construction or so. If you wanted to shave that time off, sounds like  tf.train.maybe_batch would work better. Also, you could cut down graph construction time further by reducing number of Python node creating calls by using functions. IE group together repeating parts of the graph into function nodes, and create those.

@yaroslavvb I would be very thankful if you can share some intuition/pseudo code on how to leverage tf.train.maybe_batch for this problem. Thank you!

Many thanks to @yaroslavvb : The time taken to add the operators was just 90 seconds in my case - so make_template is a reasonable approach and works great for me.
@TimZaman : Thanks a LOT for taking the time to share your approach using scopes instead of make_template - this works fine as well in my case. The latter throws an error incase of tf.Variable is used - that seems to be an advantage.

@yaroslavvb, I'd also greatly appreciate some sample code on how to use tf.train.maybe_batch. I imagine that keep_input needs to be a placeholder passed in during graph eval.
@TimZaman, I also implemented the train and val with different feeds, with two different models in different name_scopes but still reusing variables across the graphs obviously. I found that if I created the train graph first and then the val graph, evaluating the val graph still caused the train input pipeline to dequeue, implying that the graphs were not separated well enough. I see you mentioned that issue earlier in this feed. How do you ensure graph separation using different input feeds?
I created a google groups discussion to share different tensorflow workflows. I think it would be very helpful to me and other more inexperienced users if some of you guys could maybe share your code designs. Thanks!

Does anyone by now have a minimum working example using maybe_batch to switch between train and test queues? I'd greatly appreciate it!

We're planning to move away from queues and provide first-class support for multi-epoch processing in the redesigned input pipeline API. Please feel free to comment on #7951 if there are particular features that you'd particularly like to see in the new API!

in the meantime, what is the standard approach for running multiple training epochs using a queue?  I've searched around for a while, and the closest thing I could find was http://stackoverflow.com/a/39209186/212731 "http://stackoverflow.com/a/39209186/212731", for which @mrry provides a workaround. Is this the standard technique we should follow for now? or ... ?
(basically, I have a bunch of examples, which I'm happy to store in a file as tfrecords, but I need to run indefinitely; specifying the number of epochs at the start is not really ideal for me. Having to guess how many steps per epoch is also not ideal).

How would I use tf.train.maybe_batch() to select between training and testing batches?

does someone used tf.train.maybe_batch() and wants to share the way he used it ? It would be awersone

@mrry?

In my opinion, tf.train.maybe_batch() is not working properly yet. By looking at the code, it is exactly the same method as tf.train.batch(). The latter simply uses keep_input=True.
According to the API documentation, keep_input is a bool Tensor, so it should accept a tf.placeholder(tf.bool) as well, right? But when I use a placeholder, and feed it with the value True, the queue is simply blocking and nothing is happening.
I tried something like that:
is_training = tf.placeholder(tf.bool, shape=[])

...

image_batch, label_batch = tf.cond(is_training,
                                   true_fn=lambda: tf.train.maybe_batch([train_image, train_label],
                                                                        keep_input=is_training,
                                                                        batch_size=BATCH_SIZE),
                                   false_fn=lambda: tf.train.maybe_batch([train_image, train_label],
                                                                         keep_input=tf.logical_not(is_training),
                                                                         batch_size=BATCH_SIZE),
                                   strict=True)
with tf.Session() as sess:
    # initialize the variables
    sess.run(tf.local_variables_initializer())
    sess.run(tf.global_variables_initializer())

    # initialize the queue threads to start to shovel data
    threads = tf.train.start_queue_runners(coord=tf.train.Coordinator())

    for i in range(10):
        print sess.run(label_batch, feed_dict={is_training: True})
...

I thought of tf.train.maybe_batch() to be similar to the approach to use tf.cond() to switch between input-queue (for training) and feeding (for validation), but probably without having the downside described here. But it's also possible that I got this wrong...

Hi,
So any1 found a way to use tf.train.maybe_batch() appropriately ?
I am not finding out ways to switch between train/test data queues while training or eval phase.
Thanks

There is a solution using tf.FIFOQueue.from_list():
def select_batch(batches, index):
  """
  Select a batch based on the current value of the index. Only the active batch
  will be consumed. Each batch can be an arbitrarily nested tuple or list.
  """

  def _get_dtypes(tensors):
    if isinstance(tensors, (list, tuple)):
      return type(tensors)(_get_dtypes(tensor) for tensor in tensors)
    return tensors.dtype

  def _get_shapes(tensors):
    if isinstance(tensors, (list, tuple)):
      return type(tensors)(_get_shapes(tensor) for tensor in tensors)
    return tensors.shape

  def _flatten(collection):
    if isinstance(collection, (list, tuple)):
      return sum([_flatten(element) for element in collection], [])
    return [collection]

  def _unflatten(iterator, shapes):
    if isinstance(shapes, (list, tuple)):
      return type(shapes)(_unflatten(iterator, shape) for shape in shapes)
    return next(iterator)

  queues = []
  for batch in batches:
    dtypes, shapes = _get_dtypes(batch), _get_shapes(batch)
    queue = tf.FIFOQueue(10, _flatten(dtypes), _flatten(shapes))
    runner = tf.train.QueueRunner(queue, (queue.enqueue(_flatten(batch)),))
    tf.train.add_queue_runner(runner)
    queues.append(queue)
  batch = tf.FIFOQueue.from_list(index, queues).dequeue()
  return _unflatten(iter(batch), shapes)

@danijar This looks interesting. How is your provided code supposed to be used? Is batch here a single training example or a tf.train.*batch* node, or can it be both? In my case I want to enqueue many examples in varying numbers at once and dequeue them with batch_size. Furthermore, training (256x256x3) and test (512x512x3) data have different shapes. Would that work?

@tpatel0409 @LucasMahieu @danijar Have you found an example of how to use tf.train.maybe_batch in the mean time? Or did you end up switching to the new dataset api?

Personally I decided to use the "TF SLIM “ API juste before the 1.2 release : where the dataset API was introduced.
I don’t know the best solution right know.

On 29 Jun 2017, at 12:01, Julien Siems<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
@tpatel0409 https://github.com/tpatel0409 @LucasMahieu https://github.com/lucasmahieu @danijar https://github.com/danijar Have you found an example of how to use tf.train.maybe_batch in the mean time? Or did you end up switching to the new dataset api?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensorflow/issues/2514#issuecomment-311920475, or mute the thread https://github.com/notifications/unsubscribe-auth/ALc1oOEpyP9ufuOliv8aUAR7PipI2IgQks5sI3XpgaJpZM4Inwby.

@LucasMahieu Thank you for your quick response. Does the tf slim api provide any way of solving the problem? I am using tf slim as well but just for using predefined layers. I am currently looking into tf.train.maybe_batch and will post here if I come up with any example.

Maybe @ebrevdo ? Since you suggested it. It would be great if you could post a snippet

Also curious and will be reading through TF Slim data API today. I will post if I come across a solution.

https://www.tensorflow.org/versions/r1.3/programmers_guide/datasets
Looks like the dataset api is the recommended way to do it (See for example tfrecords reader). Since I am using the input pipeline I ended up evaluating batches of testdata and then feeding them back into the training queue.

How about if the labels for train and test is different? If I have two lables for train, but only one label for test.

If you are using tf.estimator.Estimator, which I think is the current best way to go (https://stackoverflow.com/questions/46925196/does-tf-estimator-estimator-train-maintain-input-fn-state), you can pass a different input_fn for train and evaluate, and you can use the mode passed to model_fn to create the appropriate graph.

I just closed this. Feel free to re-open it. In my opinion, tf.estimator and tf.dataset are the way to go, and solve my original issue.
If you need to do things at the end of each epoch then just run estimator.train() with a Dataset that does not repeat. Alternate this with estimator.evaluate() or whatever you need.
Often times the dataset is too big to wait for epoch boundaries, so use a Dataset that repeats and pass a listener to estimator.train() that runs estimator.eval() whenever a checkpoint is saved. See the link in my last comment above.
I initially had a problem with this approach because of the perceived overhead of creating the graph on each call to estimator.train() or estimator.evaluate(). But, for me, the overhead has been negligible, and  having checkpoints be the transfer of information between train() and evaluate() feels like the right approach.

Thank you! I haven't understand tf.estimator yet, I'm using the sharing weights way to do that.

@TimZaman Is there a example(make_template or share weights by sope) of try do so such things(diffrent feed for train and test). I try to use share weights method but end result that the testing data is always using the wights initialized. But the wights for traing have been updated.
Here is my code:
with tf.name_scope('train') as scope:
    dist_train = network_out(network, x_train, keep_prob= FLAGS.keep_prob, phase_train=True, batch_size=FLAGS.batch_size)

 with tf.name_scope('eval') as scope:
    with tf.variable_scope(tf.get_variable_scope(), reuse=True) as vscope:
        dist_test = network_out(network, x_test, keep_prob= FLAGS.keep_prob, phase_train=False, batch_size=FLAGS.eval_batch_size)

Any one can help me?

@markpwoodward Many thanks for your update and stackoverflow link - I would like to migrate to estimator API as well. I still have a challenge - can you share your thoughts please?
I would like to run evaluation before my epoch ends (as stated in your post as well because dataset is huge). Earlier I used to run evaluation and then save a checkpoint if score is better than the old one. Is this logical?
From your post i understand that evaluation (by a listener) is run whenever a checkpoint is saved.
What triggers a checkpoint save? How to implement the above logic?
Thank you!

@rsethur Saving only the best checkpoint would be efficient, but I am not sure how to do that with this setup. Also, evaluation loss may not be the thing you want to optimize, you might want to review a number of evaluation metrics and pick the checkpoint that looks best in a general sense. I just keep all checkpoints, and visually inspect if I need to train more.
As for setting the frequency that estimator.train() saves checkpoints, there are probably other ways to do this, but I do it in a RunConfig object passed to Estimator's constructor.
estimator = tf.estimator.Estimator(
  model_fn=...,
  config=tf.estimator.RunConfig().replace( # I'm not sure why I use replace here and not the constructor
    save_checkpoints_steps=1000, # or whatever you want
    keep_checkpoint_max=None, # defaults to last 5, but I keep all
  ),
  ...
)

Also, take a look at the new Estimator.train_and_evaluate(), you may prefer it. I still prefer my proposed approach as I actually run evaluation on a fixed subset of my training data in addition to running evaluation on my validation set. I haven't been able to get train_and_evaluate() to support this (e.g. multiple EvalSpec's)

@markpwoodward Many thanks for your detailed response - much appreciated!


[📚] Cloud Firestore to get List of Collection

Until now I haven't found any way in Documentation or Stackoverflow about how to get the list of Collections inside Firestore, like the picture below:

As you can see, I wanted to get the list of Collection inside Proyek/Publishers/{Collection}.
I have tried using this code below:
final Stream<DocumentSnapshot<Map<String, dynamic>>> pubs = FirebaseFirestore.instance.collection("Proyek").doc("Publishers").snapshots();
StreamBuilder<DocumentSnapshot<Map<String, dynamic>>>(
stream: pubs,
builder: (context, AsyncSnapshot<DocumentSnapshot<Map<String, dynamic>>> doc) {
  if (doc.hasError) {
   return SizedBox();
  } else if (doc.connectionState == ConnectionState.waiting) {
   return CircularProgressIndicator();
  } else if (doc.hasData) {
    DocumentSnapshot pub = doc.data!;
    if (pub.exists) {
      return buildPub(context, pub);
    } else {
      return Text("Empty");
    }
  }
 return SizedBox();
}, ),

  Widget buildPub(BuildContext context, DocumentSnapshot docs) {
    late String name = (docs.data()! as Map)[""];

    return Card(
      elevation: 4.0,
      child: Padding(
        padding: const EdgeInsets.fromLTRB(0, 10, 10, 10),
        child: Align(
          alignment: Alignment.centerLeft,
          child: Column(
            mainAxisAlignment: MainAxisAlignment.center,
            crossAxisAlignment: CrossAxisAlignment.start,
            children: [
              Text(name, style: TextStyle(fontSize: 20.sp)),
              SizedBox(height: 5.h),
            ],
          ),
        ),
      ),
    );
  }

The result is always empty.
Can we not get the list of Collection?
If we really can't, then what is the purpose of making a Firestore in a structure like that?
It is useless, I think.
And, there is no any documentation about this!!!

Hi @gOzaru, Thanks for filing the issue. As far as I know, firebase doesn't expose any such Api to fetch the list of collections since that is itself the top-level collection. However, I do agree this should be documented.

Hi @gOzaru, Thanks for filing the issue. As far as I know, firebase doesn't expose any such Api to fetch the list of collections since that is itself the top-level collection. However, I do agree this should be documented.

Wew.
I spent almost a week to figure this out by reading entire Firestore docs, Google and stacksoverflow.
Okay, please add it to the documentation.
So no one does the same like I did.
Thank you.
I am gonna close this.

@gOzaru  lets keep this open until this gets fixed.

It isn't possible to list the collections from any of the platforms used in FlutterFire. You can use the firebase-admin of your choice to list them however. See here


BFirebaseNetworkAdapter

Please help me to resolve this issue.

@bensmiley  @kykrueger  please help.

We recently updated the setup instructions. The correct code is:
// This is used for the app custom toast and activity transition
ChatSDKUiHelper.initDefault();

// Init the network manager
BNetworkManager.init(getApplicationContext());

// Create a new adapter
BChatcatNetworkAdapter adapter = new BChatcatNetworkAdapter(getApplicationContext());

// Set the adapter
BNetworkManager.sharedManager().setNetworkAdapter(adapter);


python312Packages.langsmith: 0.1.40 -> 0.1.42

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python312Packages.langsmith is: Client library to connect to the LangSmith LLM Tracing and Evaluation Platform
meta.homepage for python312Packages.langsmith is: https://github.com/langchain-ai/langsmith-sdk
meta.changelog for python312Packages.langsmith is: https://github.com/langchain-ai/langsmith-sdk/releases/tag/v0.1.42
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.1.42 with grep in /nix/store/5qprvh4zjkx7xanixig8k2qwc1rfk421-python3.12-langsmith-0.1.42
found 0.1.42 in filename of file in /nix/store/5qprvh4zjkx7xanixig8k2qwc1rfk421-python3.12-langsmith-0.1.42

Rebuild report (if merged into master) (click to expand)

12 total rebuild path(s)

12 package rebuild(s)

First fifty rebuilds by attrpath
khoj
pentestgpt
python311Packages.langchain
python311Packages.langchain-community
python311Packages.langchain-core
python311Packages.langchain-text-splitters
python311Packages.langsmith
python312Packages.langchain
python312Packages.langchain-community
python312Packages.langchain-core
python312Packages.langchain-text-splitters
python312Packages.langsmith

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/5qprvh4zjkx7xanixig8k2qwc1rfk421-python3.12-langsmith-0.1.42 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python312Packages.langsmith https://github.com/r-ryantm/nixpkgs/archive/5d99b874ab76d8ebbd12bee2aa99cd7fa6ab0f7f.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/5d99b874ab76d8ebbd12bee2aa99cd7fa6ab0f7f#python312Packages.langsmith

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/5qprvh4zjkx7xanixig8k2qwc1rfk421-python3.12-langsmith-0.1.42
ls -la /nix/store/5qprvh4zjkx7xanixig8k2qwc1rfk421-python3.12-langsmith-0.1.42/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

2 packages marked as broken and skipped:

khoj
khoj.dist

22 packages built:

pentestgpt
pentestgpt.dist
python311Packages.langchain
python311Packages.langchain-community
python311Packages.langchain-community.dist
python311Packages.langchain-core
python311Packages.langchain-core.dist
python311Packages.langchain-text-splitters
python311Packages.langchain-text-splitters.dist
python311Packages.langchain.dist
python311Packages.langsmith
python311Packages.langsmith.dist
python312Packages.langchain
python312Packages.langchain-community
python312Packages.langchain-community.dist
python312Packages.langchain-core
python312Packages.langchain-core.dist
python312Packages.langchain-text-splitters
python312Packages.langchain-text-splitters.dist
python312Packages.langchain.dist
python312Packages.langsmith
python312Packages.langsmith.dist

Maintainer pings
cc @natsukium for testing.

duplicate of https://github.com/NixOS/nixpkgs/pull/303002


WIP: allow mounting arbitrary NFS hosts

This is not intended to be merged - it's just a hack to allow for testing.

I think we should totally make it so if it is an efs id, we turn it into an efs hostname (not ip) and otherwise we let it pass as a normal hostname!

@rgulewich closing as this code is now in master, no need for this branch anymore (you can specify the NFS hostname directly in the EFS ID)


Updated how-to-work-on-the-component-library.md

made changes between "component" and "components" more consistent.
Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #XXXXX

Welcome to freeCodeCamp and thanks for your first PR here.

[x] My pull request has a descriptive title (not a vague title like Update index.md)

The checklist is there for a reason 😅

I went ahead and updated it to something we like and recommend.
BTW, It's all in the contributing documentation (again available in the checklist). When you follow these to the T, maintainers (not just here, elsewhere in open-source too) are more likely to take your contribution seriously.
Happy contributing.

This was my first PR, got a little bit of panic no doubt haha.
I intend to contribute frequently to freeCodeCamp, will follow your recommendation to the T.
Thanks a lot for taking the time to kindly advise me

This was my first PR, got a little bit of panic no doubt haha.

We all have been there. No worries.

I intend to contribute frequently to freeCodeCamp, will follow your recommendation to the T.

Looking forward to it.

:sparkles: :sparkles: Congratulations. :sparkles: :sparkles:
Thanks, @ZamZao, for this pull request and for contributing to the code-base for the first time.
We will be looking forward to more contributions in the future.
Cheers and happy contributing!


ElasticSearchRepository : Adding 'type' to query and search ?

Hello Quandidate,
Before making a PR : Wouldn't it be useful to add 'type' => $this->class to both the query and search functions in ElasticSearchRepository ?
My application use multiple types inside the same index (one app => one index => multiple types). And sadly i cannot directly use the methods findBy or find in ElasticSearchRepository because they lack a 'type' constraint.
I there a good reason why its not there ? If not could i make a PR for this ?
Thanx :p #

Hi,
I'm not sure, but I can't imagine it was left out on purpose. So it would make sense to add them in the query method. Also as we do already use it in the find queries.
So a PR would be appreciated :smile:
PS. It's Qandidate, not Quandidate :wink:

Thanx PR done :+1:


[INFO] BUILD FAILURE Could not resolve dependencies for project io.searchbox:jest:jar:2.4.0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:14 h
[INFO] Finished at: 2017-01-24T16:29:35+08:00
[INFO] Final Memory: 19M/157M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project jest: Could not resolve dependencies for project io.searchbox:jest:jar:2.4.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.lucene:lucene-test-framework:jar:5.5.2, org.apache.lucene:lucene-core:jar:5.5.2, org.apache.lucene:lucene-querypa
rser:jar:5.5.2, io.searchbox:jest-common:jar:2.4.0-SNAPSHOT, com.barchart.udt:barchart-udt-bundle:jar:2.3.0: Could not transfer artifact org.apache.lucene:lucene-test-framework:jar:5.5.2 from/to central (https://repo.maven.apache.org/maven2): GET request of: org/apache/lucene/lucene-test-framework/5
.5.2/lucene-test-framework-5.5.2.jar from central failed: Premature end of Content-Length delimited message body (expected: 6548748; received: 3333545 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
E:\JAVA_SRC\Jest-master\jest>
I want to know how to resolve the problem?

@laoyang360 You have the version as 2.4.0. The latest version 2 jar from Jest is 2.0.4 That's probably your issue.


MenuItem should implement EventTrigger

With the latest version of vaadin there is a support for extending MenuBar.MenuItem by a FileDownloader. This is possible because MenuBar.MenuItem is implementing the new interface EventTrigger.
This should be also supported by the MenuItems of a contextmenu.

That requires a lot of refactoring in the project, unfortunately.


Comparison with mobx-state-tree ?

Could you compare and contrast valtio-factory with mobx-state-tree (MST) ?
I'm not familiar with either, and I find it hard to discern them.
Specifically curious what you found lacking in MST which inspired you to create valtio-factory ?

Thanks for the question!

... valtio, which simplifies proxy state. think of something like mobx, now remove 99% of the api surface.
0xca0a · Nov 17, 2020

Some of the general motivation behind this library can be found in the README and a discussion post on the original valtio repository.
MST was the inspiration behind the API design but also the reason why I wrote the library in the first place. In fact, I started a React Native project based on Infinite Red's ignite template, which uses MST, and found working with it frustrating. Some of the issues I had with MST were:

too much boilerplate declaring models (especially when already using TypeScript)
even more boilerplate declaring TypeScript types for models (example)
"subscriptions" fall back on regular MobX (not part of the MST model declaration)

On the other hand, MST comes with advanced features that valtio doesn't have and it's based on MobX which has even more features. Some advanced MST features:

patches
middleware
volatile state
async actions with generators (not typesafe!)

So to summarise, I'd say MST's downsides are boilerplate, not playing too well with TypeScript, and overall complexity. However MST and MobX have more features and are more powerful than valtio and valtio-factory.
valtio-factory's strength is being a simple add-on on top of the even simpler valtio library, providing a nice sweet spot between complexity and simplicity, in my opinion.
The other pmndrs state libraries jotai and zustand are are also extremely simplistic. I recommend checking them out. Actually, some people like to organise "modern" React state management libraries into the following categories:

Flux (Redux, Zustand)
Proxy (Mobx, Valtio)
Atomic (Recoil, Jotai)

https://blog.logrocket.com/jotai-vs-recoil-what-are-the-differences

Another dimension to add would be "complexity", where valtio-factory sits somewhere between MST and vanilla valtio. I hope my answer provides a better understanding of where valtio-factory lives in this big picture and what tradeoffs you might consider.

Thanks for a fantastic  (thorough) answer! :D <3 It made a lot of sense!


Inquiry about "barriers" with Standard SSD or Standard HDD

Should we disable "barriers" for Standard SSDs or Standard HDDs with cache set to ReadOnly or None?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 978b668c-13f9-a75a-b7ca-17bb59e398a6
Version Independent ID: 8ab14f42-c192-3cde-7f94-7b481276f3d4
Content: Azure Premium Storage: Design for performance on Windows VMs
Content Source: articles/virtual-machines/windows/premium-storage-performance.md
Service: virtual-machines-windows
Sub-service: disks
GitHub Login: @roygara
Microsoft Alias: rogarana

@juyama Thanks for your comment. We are actively investigating and will get back to you shortly.

@juyama Thanks for your question. I am assigning the issue to the author to comment on this and provide requested information.

@juyama thanks for reaching out, we appreciate your situation. In this case, GitHub issues are strictly for documentation support, not product support. Answers to questions with very specific settings such as this depend heavily on your environment and use cases. If you are an enterprise customer, I suggest engaging your support team. Otherwise, if you do not have access to product support, please email<EMAIL_ADDRESS>I'll be closing this issue out now, again, thank you for reaching out.
#please-close
@VikasPullagura-MSFT FYI.


SelectField fails on zero-labeled option

Hi, Thanks for the awesome library.
Found issue similar to #214. Instead of value, set the label to 0, selectField fails.

Does this issue belong to react-autosuggest?

Sorry wrongly created, btw great library (now react-autosuggest). :wink:


Make Buffer.write* methods automatically grow capacity

Motivation:
Depending on the format, it can be hard to allocate correctly sized buffers up-front.
In these cases, it is quite convenient if the primitive write methods automatically expand the buffer until some threshold.
Modification:
Add logic to the bounds checking to automatically expand buffers when used by a primitive write method.
Result:
Buffers are now more convenient to use for writing, when the amount of data cannot be easily determined a priori.

Two things are kind of missing still:

[ ] Bulk write methods have not been updated.
[ ] The buffers will grow up to some threshold, like "max capacity", but it's currently hard coded.

wasn't the whole point of ensureWritable to have high performance and inlinable write* calls? what's the performance overhead of doing that?

@normanmaurer Still working on a meaningful concept akin to "max capacity".

@ninja- While we can't directly control inlining decisions, this logic is in the uncommon case of the bounds checks. At least for the UnsafeBuffer implementation it's not expected to regress performance, but we haven't measured yet.

@normanmaurer Add a commit for "max capacity" like concept. Please take a look at that as well.

@chrisvest please check the test failures:
[INFO]
[1441](https://github.com/netty/netty/runs/6203199775?check_suite_focus=true#step:5:1441)
Error:  Failures:
[1442](https://github.com/netty/netty/runs/6203199775?check_suite_focus=true#step:5:1442)
Error:    BufferLifeCycleTest.allocatingOnClosedAllocatorMustThrow:818 Expected java.lang.IllegalStateException to be thrown, but nothing was thrown.
[1443](https://github.com/netty/netty/runs/6203199775?check_suite_focus=true#step:5:1443)
Error:    BufferLifeCycleTest.allocatingOnClosedAllocatorMustThrow:818 Expected java.lang.IllegalStateException to be thrown, but nothing was thrown.
[1444](https://github.com/netty/netty/runs/6203199775?check_suite_focus=true#step:5:1444)
[INFO]

@normanmaurer Done


File server allows access to arbitrary files via relative paths

If we run the router example:
$ zig build example -Dexample=router

We can use curl to craft a request for any file that the user running the server can access:
$ curl "http://<IP_ADDRESS>:8080" --request-target "/files/../README.md"
# Apple Pie

Apple pie is HTTP Server implementation in [Zig](https://ziglang.org). The initial goal... (content clipped)

I'm not sure the best way to handle this situation, but two ideas off the top of my head:

Disallow paths with .. in them. Browsers (or at least Chromium and Firefox) handle this case client side, so it wouldn't interfere with browsers
Create a modified std.fs.path.resolve that keeps track of the root directory and prevents reaching beyond it

Okay, writing an algorithm to check if a path escapes was relatively simple. Hope I didn't miss any edge cases:
const std = @import("std");

fn escapes(path: []const u8) bool {
    var up: usize = 0;
    var down: usize = 0;

    // Is tokenize and not split, because we don't care about
    // empty paths (at the start/end, or between two slashes `//`)
    var component_iter = std.mem.tokenize(path, "/");
    while (component_iter.next()) |component| {
        if (std.mem.eql(u8, "..", component)) {
            up += 1;
            if (up > down) return true;
        } else if (std.mem.eql(u8, ".", component)) {
            // A single dot doesn't move us up or down
        } else {
            down += 1;
        }
    }
    return false;
}

test "preventing escaping directory with relative paths" {
    std.testing.expect(escapes("/../"));
    std.testing.expect(escapes("/./../"));
    std.testing.expect(escapes("/.././"));
    std.testing.expect(escapes("/../../"));
    std.testing.expect(escapes("/../hello"));
    std.testing.expect(escapes("/src/../src/../../hello"));
    std.testing.expect(escapes("/src/../src/.././../hello"));

    std.testing.expect(!escapes("/"));
    std.testing.expect(!escapes("/hello"));
    std.testing.expect(!escapes("/hello/.."));
    std.testing.expect(!escapes("/src/../hello"));
    std.testing.expect(!escapes("/src/../src/../hello"));
    std.testing.expect(!escapes("/src/../src/.././hello"));
}

What a great find! The file server part is still super basic, so I'm happy with you finding this now. I'll tinker a little about this and come back to it later tonight after work. I wonder if the file server should perhaps just fully disallow relative paths like that.


fix: findEpic method was searching in states entities instead of epics ones

Released in 3.2.1.
Thanks again!