All contributors

Changelog

Markserv uses Semantic Versioning
Markserv Keeps a ChangeLog

[1.17.0] - 2019-02-23
Added

Added contributors table to README. [#76]

Pull Request Test Coverage Report for Build 264

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.009%

Totals

Change from base Build 262:
0.0%

Covered Lines:
316

Relevant Lines:
433

💛  - Coveralls


Ensure all desired patches are in the final 2.7 release

Support for Mbed TLS 2.7 is scheduled to end on February 5th 2021. After releasing Mbed TLS 2.7.18 in December, we thought this might be the last 2.7.x, and stopped requiring backports for minor bug fixes that were backported to 2.16. However, since then, major issues have come up, so we will make one final release of Mbed TLS 2.7, tentatively scheduled for mid-March.
We should ensure that all desired patches are included in the final 2.7 release. This includes not only critical fixes, but also some fixes that did not warrant a release on their own, but do warrant including if we're making a release anyway.
Goals of this issue:

Ensure that all critical patches are applied: security fixes and LTS regressions.

[x] mbedtls_mpi_sub_abs buffer overflow: https://github.com/ARMmbed/mbedtls/pull/4077
[x] DRBG resource leak: https://github.com/ARMmbed/mbedtls/pull/4090

Review changes made in Mbed TLS 2.16 since the December 2.16 release (2.16.9) and ensure that everything that should be backported to 2.7 has been merged there.

Reviewed 2.16 changes since 2.16.9; created two new PRs for 2.7: #4193 and #4194


Use routers

Fixes #16 and #15 (sorry, two issues one PR bad)

Fixes #15 (sorry, two issues one PR bad)


Media Types are inconsistent

Thanks for putting together this specification. It is clear and succinct. I am confused, though, about the exact media types.
The two defined MTs are:
application/vnd.io.wasm.module.config.v1+json
application/vnd.module.wasm.content.layer.v1+wasm

However, in the example, the Media Type used for config is different:
[
  {
    "mediaType": "application/vnd.module.wasm.config.v1+json",
    "digest": "sha256:d0a165298ae270c5644be8e9938036a3a7a5191f6be03286c40874d761c18abf",
    "size": 125,
    "annotations": {
      "org.opencontainers.image.title": "runtime-config.json"
    }
  },
  {
    "mediaType": "application/vnd.module.wasm.content.layer.v1+wasm",
    "digest": "sha256:5e82b945b59d03620fb360193753cbd08955e30a658dc51735a0fcbc2163d41c",
    "size": 1043056,
    "annotations": {
      "org.opencontainers.image.title": "filter.wasm"
    }
  }
]

Which one is correct? Or am I misunderstanding the text?

For reference, in Krustket and wasm-to-oci we have been using:

application/vnd.wasm.config.v1+json  for the OCI descriptor
and
application/vnd.wasm.content.layer.v1+wasm  for the actual Wasm content layer.

We should probably investigate the OCI artifacts guidance for standardizing media types.

@technosophos thanks for opening this issue! there was a typo in the original spec which has been updated:
application/vnd.io.wasm.module.config.v1+json was from the first iteration of the spec and has been replaced by application/vnd.module.wasm.config.v1+json


Custom keybindings

Similar discussion here: https://www.reddit.com/r/neovim/comments/r77p1q/combining_luasnip_and_telescope/
When writing academic paper, it is often needed to switch between \citep (parenthesis cite) and \citet (context cite).
Sometimes, when using biblatex style, \autocite is needed.
Is there a way to customize key bindings (besides the current <cr>, C-e, C-c) so that I can add customized text into the cursor position?

Or just the cited symbol without the \cite{, so that I can customize it with my own snippets.
With the current keybindings, I always need to insert a \cite{xxx} and manually change it to \citep{xxx}.

Thanks for the reply.
I finally achieved it by using the custom format to export plain format, and use the luasnippet to create my own snippets and bind it to different keybindings.
-- Typical Abbreviations
local ls = require("luasnip")
local s = ls.snippet
local t = ls.text_node
local i = ls.insert_node

return{
  -- testing
  s("hi",  -- LuaSnip expands this to {trig = "hi"}
    {
      t("Hello, world!"),
    }
  ),

  -- citation: citep
  s("cp",
    {
      t("\\citep{"), i(1), t("}"), i(2)}, {
      callbacks = {
        [1] = {
          [events.enter] = function(_)
            return vim.cmd [[Telescope bibtex]]
          end
        }
      }
    }),

  s("ct",
    {
      t("\\citet{"), i(1), t("}"), i(2)}, {
      callbacks = {
        [1] = {
          [events.enter] = function(_)
            return vim.cmd [[Telescope bibtex]]
          end
        }
      }
    }),

  s("cc",
    {
      t(" "), i(1)}, {
      callbacks = {
        [1] = {
          [events.enter] = function(_)
            return vim.cmd [[Telescope bibtex]]
          end
        }
      }
    }),

}

That is really handy. I love the things you can do with luasnips.

Yes.
FYI, the “cc” option is for adding multiple citations into the same pair of parentheses.
This may also contribute to your future change as your current setup always add a “\cite{}” for each of the citation added through your plugin.
On 30 Mar 2023 at 13:43 +0200, Noah Wahl @.***>, wrote:

That is really handy. I love the things you can do with luasnips.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

I have implemented custom keymappings in this PR #54, a usage example can be found here: https://github.com/nvim-telescope/telescope-bibtex.nvim/tree/refactor_user_actions#custom-mappings
I would appreciate some feedback before merging this if you want to test it.
Regarding multiple citations into the same parenthesis: I think that is still limited by telescope not supporting multiple selections, but I would need to check if it is implemented by now.

Hey, thanks for the update.
Unfortunately, I don't have too much time available for setting up the testing env (changing repo, using PR, etc.). But if you merge it into the main, I can test with my current vim set up (It is easier for me).

Hi Sorry, somehow I missed the last email for testing. Just test it, it works great. Thank you for the effort.


Converted risk level transaction to task (EXPOSUREAPP-2842)

Using CWAs new task controller for risk level calculation.
Change to user experience of the app is not intended and should keep this PR from being approved.

[x]  unit tests for risk level task and RiskLevels class
[x] check changes in https://github.com/corona-warn-app/cwa-app-android/pull/1418
[x] wait for https://github.com/corona-warn-app/cwa-app-android/pull/1418 to be approved

Didn't do an indepth review yet, just short lookthrough.

i made some small changes to the risk transaction in https://github.com/corona-warn-app/cwa-app-android/pull/1418 please keep those in mind when updating this pr 👍

i made some small changes to the risk transaction in #1418 please keep those in mind when updating this pr 👍

thank you!

i made some small changes to the risk transaction in #1418 please keep those in mind when updating this pr 👍

#1418 has been merged to this branch.

It's really a pleasure not to see the old Transactions anymore! Great Work! :heart:

It's really a pleasure not to see the old Transactions anymore! Great Work! ❤

Thanks! 😃

Tested. Looks good.
But I confirm Luka's comment regarding manual mode and risk auto-update when navigating from one screen to another.

Successfully tested on Pixel (Android 10) and Pixel 5 (Android 11)


Make sorted by bar orange

Right now the sort-by indicator is red, but the sort icon is orange:

Make the sort-by indicator in the box plot the same orange too.

Done.


Bug: backspace deletes two characters on android chrome for rich text editor

Link to reproduction
Payload CMS demo
Describe the Bug
Current behaviour:
Hitting backspace in middle of content deletes two characters on Android chrome
The expected behavior:
Only the character before the cursor gets deleted
To Reproduce

Open the page or post editor on Chrome on Android
Write a word
Put the cursor somewhere in the middle
hit backspace - two characters get deleted. One character from front and one from back

Payload Version
2.1.0
Adapters and Plugins
No response

@kirandash are you using the lexical or slate RTE?

@kirandash are you using the lexical or slate RTE?

@JarrodMFlesch thanks for the response!
I am using lexical. You can reproduce the issue on the demo link too: https://demo.payloadcms.com/

FYI, this is a lexical issue and as pointed out by lexical, the root cause is the issue from android chrome.
https://github.com/facebook/lexical/issues/4941
But I am curious if you guys have a patch for this. Since the issue from lexical is there since almost a year and no fixes/patch is known ASFAI

Looks like this was patched by lexical recently! Once the new lexical version is out, I'll update it. Closing this for now, as this mainly seems to be a lexical/chromium issue rather than an issue with payload's implementation.


termscp: 0.15.0 -> 0.16.0

Diff: https://github.com/veeso/termscp/compare/refs/tags/v0.15.0...v0.16.0
Changelog: https://github.com/veeso/termscp/blob/v0.16.0/CHANGELOG.md

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 349034

x86_64-linux

:white_check_mark: 1 package built:

termscp

Needs to be updated to this commit to fix the build failure. https://github.com/veeso/termscp/commit/e45c3d5b4ef64653e5b6cc4f3703e3b67514306d

Superseded by https://github.com/NixOS/nixpkgs/pull/355461


Update confirmations parser

Maybe I am wrong but now steam uses checkbox to confirm operations on mobile. Current parser cannot handle this for now:
PHP Notice 'yii\base\ErrorException' with message 'Undefined offset: 1'

in \vendor\waylaidwanderer\php-steamcommunity\MobileAuth\Confirmations.php:53

Closed in #77


[Feature Request] LockManager uninstall procedure

Is your feature request related to a problem? Please describe.
I was using LockManager with ZHA then with OpenZwave and now I'm going to ZWaveJS. As LockManager became KeyMaster I would like to redo things from scratch. Is there any uninstall procedure ? What to delete, how to get rid of all the entities which have been created by LockManager before begining with KeyMaster. And...
Just a word about renaming entities, as some default lock names are very long, putting a suffix as the lock name some times (at least on my screen which is not big) doesn't help, I mean only the first characters are seen on the interface which doesn't help choosing from lock.be469zp_connect_smart_deadbolt_locked_back_door and lock.be469zp_connect_smart_deadbolt_locked_front_door as the interface only shows lock.be469zp_connect_smart_deadbo... Wouldn't it be better to make it a prefix ?
Describe the solution you'd like
1- An LockManager uninstall procedure in the WiKi
2- Prefix instead of suffix in the entities renaming
Describe alternatives you've considered
Additional context
I suppose that is because of my small screen as I didn't see anyboby else complain about this naming problem.

Prefix instead of suffix in the entities renaming

Renaming isn't required, it's optional and for your sanity.

An LockManager uninstall procedure in the WiKi

This is a good idea, we'll see what we can do.

So I removed the integrations and uninstalled it through HACS and rebooted. After that, I then removed entities that said Keymaster and the automations related to Keymaster
I rebooted again, reinstalled through HACS, then rebooted again, but I do not see Keymaster in the integrations. Would this be a separate issue or related to the uninstall process?

Press F5 in the integrations screen, then try to add it.

Press F5 in the integrations screen, then try to add it.

That was exactly it, thanks for that!

After uninstalling the integration, then removing through HACS and doing several reboots, I am left with a LOT of entities still. How do I remove all of these?

remove any files related to keymaster in your packages folder, typically config/packages, then reload all

Thanks for the reply. I've removed the files from /config/packages (the packages dir) and reloaded all the automations, restarted HA, etc, but all of these entities are still around.
I've also tried a lot of other things, reinstalling it in HACS, reinstalling it as an automation, disabling/enabling keymaster of an automation, with lots of restarts in between.
Maybe I'm doing this wrong. Can someone give more detailed instructions? For example I don't see a "reload all automations" button AFAICT.

I ended up just resolving it manually by fully uninstalling the add-on, then removing keymaster from HACS, removing everything from the /config/packages/ dir, and then restarting everything. Then I went to /config/entities and ended up manually removing all the entities, helpers, etc associated w/ he keymaster install.
I needed to reinstall with another lock, so I basically then did a fresh install and I was all set.

I have exactly the same problem - having deleted the integration for my lock, then deleting keymaster from HACS, and having done several re-starts, I am still left with loads of keymaster automations.   And it seems I cannot delete them because  “Only automations in automations.yaml can be deleted” and these aren’t.
Using File Editor I can’t see the /config/entities folder that @bmbouter refers to.   Is there anywhere else to look or other ways to tidy this up?

Using File Editor I can’t see the /config/entities folder

They were referring to the menus in HA. Check /config/packages, but the integration should nuke all the helpers it created on removal.

I have exactly the same problem - having deleted the integration for my lock, then deleting keymaster from HACS, and having done several re-starts, I am still left with loads of keymaster automations. And it seems I cannot delete them because “Only automations in automations.yaml can be deleted” and these aren’t.
Using File Editor I can’t see the /config/entities folder that @bmbouter refers to. Is there anywhere else to look or other ways to tidy this up?

If you are still having trouble with this I solved it for me by adding keymaster back and recreating the locks and name them exactly like they were before. So if you look at your helpers and they are named some like xxxxxxxx.front_door make sure you name them with a space when recreating them. I kept trying with frontdoor and forgot they had a space the first time. Once they are recreated and the files are generated I was able to remove them from the devices & services and all the helpers were deleted.

Same issue for me.  I installed Keymaster and set up 4 locks with one of them as the parent of the other three.  After removing all four locks from the integrations, then removing Keymaster from HACS I'm letft with about 1,500 unavailable entities.  I've even installed Home Assistant Registry in HACS in order to use the ha_registry.remove_entity service.  But the task of individually selecting all these entities from the dropdown list seems outrageous...  Any help?

Same issue for me. I installed Keymaster and set up 4 locks with one of them as the parent of the other three. After removing all four locks from the integrations, then removing Keymaster from HACS I'm letft with about 1,500 unavailable entities. I've even installed Home Assistant Registry in HACS in order to use the ha_registry.remove_entity service. But the task of individually selecting all these entities from the dropdown list seems outrageous... Any help?

By the way, I followed mattbruman's process above.  I re-installed Keymaster, naming the locks just as before.  The number of unavailable entities went way down.  Clear down to less than 100.  But after removing the locks from the integration I was back up to 1,500+ unavailable entities.

Same issue for me. I installed Keymaster and set up 4 locks with one of them as the parent of the other three. After removing all four locks from the integrations, then removing Keymaster from HACS I'm letft with about 1,500 unavailable entities. I've even installed Home Assistant Registry in HACS in order to use the ha_registry.remove_entity service. But the task of individually selecting all these entities from the dropdown list seems outrageous... Any help?

By the way, I followed mattbruman's process above. I re-installed Keymaster, naming the locks just as before. The number of unavailable entities went way down. Clear down to less than 100. But after removing the locks from the integration I was back up to 1,500+ unavailable entities.

I’m sorry, no bright ideas from me. I tried the reinstall/deinstall to nuke things but it didn’t work. I ended up just deleting each entity in batches of 50 or so until they were gone, but there were some that I couldn’t get rid of as they had no entity ID (sorry, can’t quite remember the exact error message) so I’ve lived with those.

Just commenting here that I read the directions FIRST and went to the integration and removed the locks via the integration screen, I still was left with a bunch of broken entities no longer provided by the integration and I am now picking them out of my HA install.


virtual doesn't get snatpool config

Environment

Application Services Version: 3.22.1
BIG-IP Version:<IP_ADDRESS>

Summary
declaration tries to push a snatpool to a virtual with the Service HTTPS template. The response to the declaration is:
"code": 200,
"message": "no change",
and the virtual continues to have automap.
Steps To Reproduce
Steps to reproduce the behavior:

Submit the following declaration:

{
   "class": "AS3",
   "action": "deploy",
   "persist": true,
   "declaration": {
      "class": "ADC",
      "schemaVersion": "3.0.0",
      "id": "abc123",
      "label": "labels the declaration",
      "updateMode": "selective",
      "remark": "another comment field",
      "as3-partition": {
         "class": "Tenant",
         "defaultRouteDomain": 110,
         "as3-app": {
            "class": "Application",
            "template": "generic",
            "as3-vs": {
               "class": "Service_HTTPS",
               "virtualAddresses": [
                  "<IP_ADDRESS>/32"
                ],
               "virtualPort": 8443,
                "profileHTTP": {
                    "use": "as3-http"
                },
                "iRules": [
                    {
                    "bigip": "/Common/hoc_errorpage_ir"
                    }
                ],
                "SNATPool": {
                    "bigip": "/Common/SNAT-DCLB-Legacy"
                },
               "serverTLS": "as3-client-ssl",
               "clientTLS": "as3-server-ssl",
               "pool": "web_pool"
            },
            "as3-http": {
                "class": "HTTP_Profile",
                "xForwardedFor": true
            },
            "as3-client-ssl": {
                "class": "TLS_Server",
                    "certificates": [
                    {
                        "certificate": "as3-cert"
                    }
                ]
            },
            "as3-cert": {
                "class": "Certificate",
                "remark": "in practice using a passphrase is recommended",
                "certificate": {
                    "bigip": "/Common/default.crt"
                },
                "privateKey": {
                    "bigip": "/Common/default.key"
                }
            },
             "as3-server-ssl": {
                "class": "TLS_Client",
                "label": "simplest decl requires just cert"
            },
            "web_pool": {
               "class": "Pool",
               "monitors": [
                  "http"
               ],
               "members": [{
                  "servicePort": 80,
                  "serverAddresses": [
                     "<IP_ADDRESS>",
                     "<IP_ADDRESS>"
                  ]
               }]
            }
         }
      }
   }
}

Observe the following error response:

{
    "results": [
        {
            "code": 200,
            "message": "no change",
            "host": "localhost",
            "tenant": "as3-partition",
            "runTime": 1623
        }
}

Expected Behavior
Have the virtual updated with the snatpool
Actual Behavior
No change and no error. virtual remains configured with automap.

@danpacheco1 Unfortunately you are encountering a quirk of our JSON schema, so AS3 is failing to inform you that your declaration is incorrect. The property you are looking for is snat, not SNATPool.

I just modified the declaration with snat instead of SNATPool and it works.
Thanks @dstokesf5


[RemoteRendering] Add missing await

This also passes mypy 1.0.0 check.

API change check
API changes are not detected in this pull request.


agi: install icons to the right directory

Description of changes
Fixes a regression from #191395
Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and “core” functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[X] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 191488 run on x86_64-linux 1

1 package built:

agi

Thank you!


Update README.md

Just a writing fix.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I don't understand the reasons on the Codacy/PR Quality Review about spaces, since I only added the letter n to the word Aother, making it Another

I wouldn't worry about Codacy as we are dropping that tool soon. Thanks for the PR


Interoperability problem (redux-form and react-dropdown): New autofocus field does not get focus

Are you submitting a bug report or a feature request?
Bug report, provided that my implementation isn't faulty.
What is the current behavior?
I can't get react-dropdown (1.5.0) and redux-form (7.4.2) to work together in this case:

My redux-form first only has one react-dropdown
When a value is selected in the dropdown, a new redux-form field with autofocus is added to the form

Current behavior: The new field does not have focus after being added to the form.
What is the expected behavior?
The new field has focus.
Sandbox Link
See example at https://codesandbox.io/s/7ow3l2yn7j .
What's your environment?
Chrome 67 on Windows.
Other information
The focusing works as expected if I implement the dropdown using the approach in the redux-form Simple Form Example. The difference is shown in the sandbox (changing useReactDropdown value in form.js toggles between using/not using react-dropdown). Also reported this as react-dropdown #135, as I'm unsure on the source of the problem.

This report hasn't isolated an issue with redux-form which is what the issue tracker is for. Please use Stack Overflow / Reactiflux for general questions. I'd suggest spying on focus events to debug. It's possible react-dropdown is messing with focus after an onselect event.

@danielrob, thanks for the suggestion. In case someone stumbles on this at a later date, I've opened a Stackoverflow question with some additional observations.


update canCommitGrandparent logic to avoid underflow

Current dependencies on/for this PR:

feature/pos-fast-hotstuff-consensus

PR #742

PR #750   👈

This comment was auto-generated by Graphite.


Probably static.copy/static.paste would be good

It would be nice to be able to access copy/paste data from Lua. SDL2 supports copy/past reading iirc

See my auto-generatinng code hooks for C++ to get started


Mod packaged improperly

          I left a comment on ESO.UI. The files aren't in their own "folder" when installing through Minion. The LibRandomMount, Localization folders and the manifest and LUA files get installed in the addons folder. ESO doesn't recognize the addon as being installed. I have moved my files in to a folder named RandomMount.

I also think that having the extra API versions in the manifest are making it state that it's out of date. If you change that text file to say<PHONE_NUMBER>38 it removers the "out of date" in the addons list.
Originally posted by @sshogrin in https://github.com/Oricul/eso-RandomMount/issues/6#issuecomment-1604303115

I've corrected this and re-uploaded 3.7.

It's also not pulling out the random mount when you have a companion out or when grouped.


RVIZ crashes on loading default.rviz

RVIZ suddenly crashes on loading ros/src/.config/rviz/default.rviz. in step 3.2 of the QuickStart demo. The messages left were:
[ INFO] [1483705468.774296510]: rviz version 1.11.15
[ INFO] [1483705468.774336451]: compiled against Qt version 4.8.6
[ INFO] [1483705468.774345900]: compiled against OGRE version 1.8.1 (Byatis)
[ INFO] [1483705469.009310064]: Stereo is NOT SUPPORTED
[ INFO] [1483705469.009366250]: OpenGl version: 3 (GLSL 1.3).
['sh', '-c', '$(rospack find runtime_manager)/../../../.config/rviz/cmd.sh']
pid=15636
sched policy=OTHER prio=0
pid=15637
sched policy=OTHER prio=0

(rviz:15644): GLib-CRITICAL **: Source ID 521 was not found when attempting to remove it

(rviz:15644): GLib-CRITICAL **: Source ID 539 was not found when attempting to remove it

(rviz:15644): GLib-CRITICAL **: Source ID 556 was not found when attempting to remove it

Please let me know how to solve the problem.
Steps to repro the crash
I cleanly installed Ubuntu 14.04 on an amd64 machine and set up Autoware by following steps. The machine has a Geforce GT 420 graphic card but I didn't install CUDA.
Firstly I installed OpenCV 2.4.13 from source and also installed GLEW and Qt5 by:
sudo apt-get update
sudo apt-get install libglew-dev qt5-default

Installing ROS and dependencies for Autoware,
sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu trusty main" > /etc/apt/sources.list.d/ros-latest.list'
wget http://packages.ros.org/ros.key -O - | sudo apt-key add -
sudo apt-get update
sudo apt-get install ros-indigo-desktop-full
sudo rosdep init
rosdep update

sudo apt-get install ros-indigo-nmea-msgs ros-indigo-nmea-navsat-driver ros-indigo-sound-play ros-indigo-jsk-visualization
sudo apt-get install libnlopt-dev freeglut3-dev qtbase5-dev libqt5opengl5-dev libssh2-1-dev libarmadillo-dev libpcap-dev gksu libgl1-mesa-dev

and I successfully built Autoware.
Then, I executed steps 2.1--3.2 following QuickStart.
On the other hand, I also set up the same environment on a Hyper-V virtual machine; RVIZ successfully opened the config file in this environment.

Sorry. I used not Geforce GT 420 but Geforce 740.

I know the same crash.
There are cases to solve by playing rosbag a little.
I think default.rviz has the problem.

Thank you for your reply.
I reinstalled Ubuntu14.04 and Autoware and ran RVIZ after playing rosbag to 5%. However, RVIZ crashed again.
Log messages obtained were:
['sh', '-c', '$(rospack find runtime_manager)/../../../.config/rviz/cmd.sh']
pid=17531
sched policy=OTHER prio=0
pid=17532
sched policy=OTHER prio=0
/home/xxx/Autoware/ros/src/util/packages/runtime_manager/../../../.config/rviz/cmd.sh: line 44: 17539 Segmentation fault      rosrun rviz rviz

I  got the same crash.  When I tried 'rosrun rviz rviz'   in another terminal, specified the 'default.rviz'. The same crash would happen again. Here's the info:
$ rosrun rviz rviz
[ INFO] [1489087826.288732430]: rviz version 1.11.15
[ INFO] [1489087826.288785230]: compiled against Qt version 4.8.6
[ INFO] [1489087826.288797872]: compiled against OGRE version 1.8.1 (Byatis)
[ INFO] [1489087827.048216091,<PHONE_NUMBER>.673704139]: Stereo is NOT SUPPORTED
[ INFO] [1489087827.048296199,<PHONE_NUMBER>.673704139]: OpenGl version: 4.5 (GLSL 4.5).
Segmentation fault (core dumped)
I haven't figured  out a fix, am doing search on 'rosrun rviz rviz, segmentation fault'

Confirmed. Autoware comes up successfully, RViz launches and then immediately crashes upon loading default.rviz.
Help would be appreciated!

I have met this problem too. And I got a resolution like this:
Change the last two lines of $path/Autoware/ros/src/.config/rviz/default.rviz, which like:
X: 1895
Y: 44
to smaller numbers:
X: 100
Y: 100
save the file. And load it from rviz again. Wish this could help you.

I have the same problem

Did you guys find a solution to this problem? I have the same problem!

I got the same crash. When I tried 'rosrun rviz rviz' in another terminal, specified the 'default.rviz'. The same crash would happen again. Here's the info:
$ rosrun rviz rviz
[ INFO] [1489087826.288732430]: rviz version 1.11.15
[ INFO] [1489087826.288785230]: compiled against Qt version 4.8.6
[ INFO] [1489087826.288797872]: compiled against OGRE version 1.8.1 (Byatis)
[ INFO] [1489087827.048216091,<PHONE_NUMBER>.673704139]: Stereo is NOT SUPPORTED
[ INFO] [1489087827.048296199,<PHONE_NUMBER>.673704139]: OpenGl version: 4.5 (GLSL 4.5).
Segmentation fault (core dumped)
I haven't figured out a fix, am doing search on 'rosrun rviz rviz, segmentation fault'

Hi zzkou:
Did you have the solution for this issue ? I have the same issue like yours. Thanks in advance !


Data Flow: Navigate View History (Past | Present | Future)

Description
One of the most common use cases when visualising data is the ability to move forward and back over different states of the data. For example: go back to where the user was in the previous step.
fiscal-data-js provides an API for accessing history (previous states, current state, future states). This API needs to be integrated with the view components in order to allow such a user flow.
Tasks

[x] Integrate fiscal-data-js history states
[x] Write tests for the integration

DONE.


Tab Confusion

Issue Type: Feature Request
The current tab is indistinguishable from other tabs. I find myself closing all tabs so I know where I am.
Please change the appearance of the current tab.
Azure Data Studio version: azuredatastudio 1.26.1 (796abbdf5fdd341e3528e30a6b777c93e115eb02, 2021-02-24T23:42:55.085Z)
OS version: Windows_NT x64 10.0.19041

@glennoleum Thanks for opening the issue. The current tab always has a darker background compared to other tab backgrounds. Is there something else you're suggesting here?

closing, no reply after 7 days. @glennoleum please reopen when you getting a chance to reply to @abist 's question above.


Version 2.9.1 is reported as vulnerable

Describe the bug
JetBrains Rider signal the 2.9.1 version as vulnerable.
It doesn't provide details on the motivation.
This is also mentioned here:
https://github.com/jeremylong/DependencyCheck/issues/6048
https://github.com/jeremylong/DependencyCheck/issues/6088
Unfortunately some of the projects/customers I'm working on/with refuse to use libraries with known vulnerabilities.
Is there a mitigation ?
Many thanks in advance for any suggestion.
To Reproduce
Just add the NuGet package to any project in Rider, and the warning will appear
Expected behavior
No vulnerabilities signaled
Screenshots

With the Version 2.9.2-ci-210:

JetBrains Rider does not report any vulnerability 😃

Thankyou @schittli :)
When will this version be available on NuGet?
I don't see it listed on NuGet.org yet.

Thankyou @schittli :) When will this version be available on NuGet? I don't see it listed on NuGet.org yet.

Any update here? Nuget.org still has 2.9.1


Lightweight CLEOS client

I'm trying to implement a lightweight client based on cleos that does only read commands such as "get", "wallet", transfer and version without requiring the client to install all 20Gb of EOS.IO.
I'm working with curl, but is there a better way to build this?

Curl and/or python are probably good ways to do this.


ValueError: embedded null character

Python: 3.7.15 x64
xdialog: <IP_ADDRESS>
Error msg:
  File "C:\ProgramData\Miniconda3\envs\test3.7\lib\site-packages\xdialog\__init__.py", line 66, in open_file
    return dialogs.open_file(title, filetypes, multiple)
  File "C:\ProgramData\Miniconda3\envs\test3.7\lib\site-packages\xdialog\windows_dialogs.py", line 53, in open_file
    opts.lpstrFilter = ''.join(out)+'\0' # Extra NULL byte just in case
ValueError: embedded null character

I tried to solve this problem myself using the solution from here:
https://github.com/arsenetar/send2trash/issues/17
https://github.com/arsenetar/send2trash/pull/18/commits/213417d3f2562f0dd5c9e11bdca272b6fb355fed
And I was able to fix it, but it looks pretty rough and only for this function (xdialog.dialogs.open_file).

This is kinda dumb, but alright lol

Fixed by https://github.com/xMGZx/xdialog/commit/c4e13fad294b3a054e8fe5294d03e1146108169a


[EIP-1559] Add a new gas pool available for processing eip-1559 transactions

Description
As per specified in the EIP, we must have 2 different gas pools during the phase 1 where both legacy and new transactions are accepted.
AbstractBlockProcessor.processBlock should compute the gas budget for a transaction accordingly.

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/mainnet/AbstractBlockProcessor.java
processBlock method
final long remainingGasBudget = blockHeader.getGasLimit() - gasUsed;
remaining gas budget should be computed according to EIP-1559 rules


Documentation

Add a few more commands to remove ambiguity and added instructions as to how to use the application

Proposed fix for #3 , Please ignore the earlier commit and check the latest commit alone.

@Solai-Adithya please make the requested changes. Keep in mind that we need to keep the README as simple as possible.
Just as a precaution from the next time, make PRs from a separate branch of your fork to avoid merge conflicts :smile:


support xdp auth in tailcall

What type of PR is this?

What this PR does / why we need it:
support xdp auth in tail call
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

/retest

/retest

/retest

/retest

/retest

/retest

/retest

A switch needs to be added to determine whether to enable link-level authentication or packet-based authentication.

It is painful for me to review such a big pr  with round and round update/review. We should incrementallly update. Never force push


Measurable Relationships: allow bulk selection

use tree control with checkboxes.

Investigate difficulty

Some thoughts:

If a measurable in the tree is the same as the parentRef it should be marked non-concrete to prevent it being selectable

To add relationships the control needs to track already existing relationships and create relationships only for newly checked boxes.  Additionally, the relationship and description needs to be added, should this be shared or explicitly required for each selection?

For deletion, there can be several relationships backing a specific checkbox entry since direction is lost when representing these relationships on the tree (side a and side b).  How to ensure only the intended relationship is removed?

show existing relationships and checked and disabled, so you can only delete through the existing table layout
same as above except the table layout is only shown when a node is clicked at which point the user can delete a node (similar to ratings editor)

a) agree
Thought we were moving away from checkboxes to a master-detail style interaction....
b) simply becomes a styling issue - we reload the entire tree data on save so no tracking reqd
c) avoiding checkbox tree simplifies this I think
Not sure what 1) and 2) are getting at.   2) sounds more like what we discussed....

2 possible options below.

think option 2 sounds like the nicer user experience

Something like this?


Fix error reading TIFF input

In PointCloud::TransformPointCloud, set the PDAL reader type explicitly
when the input file is TIFF. This is necessary because the GDAL reader
plugin doesn't specify any file extensions [1], and therefore the reader
cannot be inferred based on the filename.
This fixes errors like the following when running align3d:
Writing aligned LAS file.
Cannot determine reader for input file: /path/to/DATASET.tif

[1] https://github.com/PDAL/PDAL/blob/58b586fb392085a0ea32dcdedc488f78d7ea2c00/io/GDALReader.cpp#L46-L49

Closing, as the generated file is not used by CORE3D metrics.


Fix Hermes sourcemaps for custom build variants

Summary
It's a common use case to have a custom buildType on android that should be similar to release for staging builds
For those, due to this line there is no easy way to make sure the sourcemaps are output.
This leads to issues such as https://github.com/getsentry/sentry-react-native/issues/1105
At the moment, this is one of the blockers to use Hermes on our side, and oh boy do we want to use it! Congrats on that, it's quite fantastic!
One option would be to add a flag specific for Hermes, like useHermesReleaseConfigIn, but then that would be an additional flag to set to true in addition to bundleIn and devDisabledIn. You'd have something like:
project.ext.react = [
        entryFile        : "index.js",
        bundleInBeta     : true,
        devDisabledInBeta: true,
        useHermesReleaseConfigIn: true,
        enableHermes: true,  // clean and rebuild if changing
]

My suggestion is then to have one flag to rule them all for such variants so that you only have to add:
project.ext.react = [
        entryFile        : "index.js",
        useReleaseConfigInBeta: true,
        enableHermes: true,  // clean and rebuild if changing
]

I started by refactoring and moving all uses of targetName.toLowerCase().contains("release")  into a single variable, then I introduced the new flag.
Of course, open to any challenges/feedbacks!
Changelog
[Android] [Added] - Add gradle flag to enable all release config
[Android] [Fixed] - Allow Hermes sourcemaps for custom build types
Test Plan

I have a project with 3 build types:

    buildTypes {
        debug {
            applicationIdSuffix ".debug"
            signingConfig signingConfigs.debug
        }

        beta {
            applicationIdSuffix ".beta"
            signingConfig signingConfigs.beta
            matchingFallbacks = ['release']
        }

        release {
            minifyEnabled enableProguardInReleaseBuilds
            proguardFiles getDefaultProguardFile("proguard-android.txt"), "proguard-rules.pro"
            signingConfig signingConfigs.release
        }
    }

I've replaced my config which was previously:

project.ext.react = [
        entryFile        : "index.js",
        bundleInBeta     : true,
        devDisabledInBeta: true,
        enableHermes: true,  // clean and rebuild if changing
]

by
project.ext.react = [
        entryFile        : "index.js",
        useReleaseConfigInBeta: true,
        enableHermes: true,  // clean and rebuild if changing
]

I've built all variants of the app:

./gradlew clean assembleRelease
./gradlew clean assembleBeta
./gradlew clean assembleDebug

[x] All 3 builds now work. assembleBeta was not working because we use Sentry for crash reporting, and it wasn't able to find any sourcemaps.
[x] On Beta and Release sourcemaps are now correctly uploaded to Sentry and the app runs fine.
[x] On Debug, the app runs fine, the bundle task was not run, dev menu and reload work as expected

Hi @cpojer, I'm pinging you since you seem to be the last person who reviewed PRs on react.gradle, but let me know if that's not appropriate!
Would love to get some feedback on this one if possible! 🙏

Thanks for sending the PR. I'm closing this as the logic is completely rewritten now and includes this logic you suggested


Feat: create RadioGroup component

Closes #29

:tada: This PR is included in version 0.19.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix tests on GitHub Actions

Tests run locally work just fine on macOS, Windows, and Ubuntu, but fail on GitHub Actions. Likely because of a fs.writeFileSync issue with Node in that environment.

Fixed in v2.9.1


Module 'openai' has no attribute 'Embedding'

System Info
LangChain: 0.0.331
OpenAI: 1.1.0
Python:  3.11.2
Who can help?
@hwchase17 @agola11
Information

[ ] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[X] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
Full Message:

...langchain/embeddings/openai.py", line 284, in validate_environment
values["client"] = openai.Embedding
^^^^^^^^^^^^^^^^
AttributeError: module 'openai' has no attribute 'Embedding'. Did you mean: 'embeddings'?

Code line caught this error:
index = VectorstoreIndexCreator(vectorstore_kwargs={"persist_directory":"persist"}).from_loaders([loader])
Expected behavior
Creates Vector Store Index

I am also having the same issue.  Dropped back several version of openai library to no avail.  I am using python 3.11.4.  Latest openai (1.1.1) and langchain 0.0.331.  I also attempted version 0.0.330 of langchain and still getting the same issue.  The openai library seems to use openai.embeddings instead of openai.Embedding.

I'm also having the same issue with Python 3.11.4, openai (1.1.1), and LangChain 0.0.331.

It looks like openai-python restructured package at their v1.0.x release https://github.com/langchain-ai/langchain/issues/12967#issuecomment-1797092130. This is likely the root cause that langchain can't find a designated class in openai package namespace.
We need to find an alternative class in the new architecture.

I'm also having the same issue with Python 3.11.4, openai (1.1.1), and LangChain 0.0.331.

Temporal "fix".. pin openai to 0.28.1 version! @alexmc2
openai==0.28.1

It's working! Thanks so much! :D

I'm also having the same issue with Python 3.11.4, openai (1.1.1), and LangChain 0.0.331.

Temporal "fix".. pin openai to 0.28.1 version! @alexmc2
openai==0.28.1

It's working! Thanks so much @dosuken123! :D

Having this same issue as well, trying the fix

I'm also having the same issue with Python 3.11.4, openai (1.1.1), and LangChain 0.0.331.

Temporal "fix".. pin openai to 0.28.1 version! @alexmc2
openai==0.28.1

Life Saver !!

I'm also having the same issue with Python 3.11.4, openai (1.1.1), and LangChain 0.0.331.

Temporal "fix".. pin openai to 0.28.1 version! @alexmc2
openai==0.28.1

Thanks, it worked for me.

I'm also having the same issue with Python 3.11.4, openai (1.1.1), and LangChain 0.0.331.

Temporal "fix".. pin openai to 0.28.1 version! @alexmc2
openai==0.28.1

Thank you, it worked for me.

I'm also having the same issue with Python 3.11.4, openai (1.1.1), and LangChain 0.0.331.

Temporal "fix".. pin openai to 0.28.1 version! @alexmc2
openai==0.28.1

@benjaminbascary that worked. thanks!

I have the same issue in Colab

same issue on collab, even with openai==0.28.1..

It works with openai==0.28.1, but need to restart kernel after re-install

Many thanks 🙏
On Wed, 8 Nov, 2023, 4:27 am Seifeur guizeni, @.***>
wrote:

same issue on collab, even with openai==0.28.1..
—
Reply to this email directly, view it on GitHub
https://github.com/langchain-ai/langchain/issues/12954#issuecomment-1800726369,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUS2GB6HS7IH6SHTYHCFT2LYDLGXDAVCNFSM6AAAAAA7AFKVDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBQG4ZDMMZWHE
.
You are receiving this because you commented.Message ID:
@.***>

Yes openai = "0.28.1" solves the problem thanks!

is there any update - want to use this with new version of openai so i can use the latest models

Solved in the latest update.

Thanks
On Sun, 12 Nov, 2023, 3:43 am Seifeur guizeni, @.***>
wrote:

Solved in the latest update.
—
Reply to this email directly, view it on GitHub
https://github.com/langchain-ai/langchain/issues/12954#issuecomment-1806945457,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUS2GB6MEYI2NWG4WPQT2V3YEAESHAVCNFSM6AAAAAA7AFKVDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWHE2DKNBVG4
.
You are receiving this because you commented.Message ID:
@.***>

openai==0.28.1

thank you!!!!!!!!!!!!
my  langchain Version: 0.0.316
openai==0.28.1
works for me too.

It works for me too


Issue358 pbh

@csmoore please review and merge

@csmoore please review

@csmoore one last time....

@csmoore -- removed unused variable and fixed backspace issue please review...


change error format

fixes https://github.com/omgnetwork/ex_plasma/issues/57
I grepped elixir-omg. it seems it doesn't use the old name

ugh good catch. why didn't dialyzer catch this?

@InoMurko it wasn't the wrong type definition issue. This PR changes naming for exceeds_maximum_value => cannot_exceed_maximum_value

Now I'm curious. Where did you notice cannot_exceed_maximum_value? I can't find it

@InoMurko Arthur suggested in the description of the issue
Currently, we use a few errors that follow the pattern of cannot_something_something. We should change errors like exceeds_maximum_value to cannot_exceed_maximum_value or something

https://github.com/omgnetwork/ex_plasma/issues/57

@mederic-p ping


subscribe() / publish() is done from main-thread, not connection-thread

subscribe() / publish() is done from main-thread, not connection-thread. When the send-buffer is full, this causes the main application to block, while the functions are mentioned to be non-blocking.
Hand-off the request to the connection-thread for transmission. This increases latency slightly, but in return the function becomes truly non-blocking.

After some consideration, went for a different approach.
The socket is now non-blocking, which means send returns when the sndbuffer is full. To avoid copying memory and wasting CPU, outgoing packets are done from the thread the publish is called from. This will never block, as the send called is non-blocking.
However, if the sndbuffer is full, publish() will start returning false, as it couldn't actually transmit the message. It is up to the user of this library to think what to do about this.
This way, we avoid complexity in the library will keeping a lot of speed.
The subscribe handling is done from the connection thread, mostly as that is much more practical. However, this means that the callbacks should be as quick as possible to efficiently handle a lot of incoming messages.
This balance between what-to-send-in-what-thread feels much better, and is very quick.

Nope, I am wrong, and this fails.
Mainly, the problem ofc is if the sndbuf isn't big enough to fit your message in. In that case, you need to do partial sends, and wait till you can actually transmit the whole packet. That makes all of the above non-sense.
So, we are back to: we need to offload this to another thread. Originally I was thinking doing this in the connection thread, but that is nearly impossible. Instead, it should be done in yet-another-thread (that means every connection has 1 read and 1 write thread).
Additionally, and this truly is a bug, currently the send result is not checked how much data was actually put on the socket.

Implemented in a980dab1859014f040c09db715b2a4918785d53f


[HOLD on #21498] Web -  App allows to modify announce, admin room name with deep link

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open the app
Open settings
Open workspaces
Open any workspace
Click on 3 dot and click on go to #announce room
Click #announce in head to open details page
Click on settings and observe that it does not allow to update room name
Add '/room-name' to URL and click enter, observe that it opens room name edit page with name as # announce

Expected Result:
App should not allow users to open announce and admin rooms room name edit page using deep link
Actual Result:
App allows users to open announce and admin rooms room name edit page using deep link
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.41-2
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/93399543/40b9babc-b98d-4f69-b7de-e8f792c27d9b
https://github.com/Expensify/App/assets/93399543/c6df550d-078c-44e9-a6f0-d5dd3190dfac
Expensify/Expensify Issue URL:
Issue reported by: GH handle - @dhanashree-sawant
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1689410765634399
View all open jobs on GitHub

Proposal
Please re-state the problem that we are trying to solve in this issue.
Web - App allows to modify announce, admin room name with deep link.
What is the root cause of that problem?
We don't have any check for validating that room's name can be changed in RoomNamePage.
What changes do you think we should make in order to solve the problem?
We shall add the check in RoomNamePage similar to we did in RoomSettingsPage
we can use this logic -
this.shouldDisableRename(linkedWorkspace) || ReportUtils.isChatThread(this.props.report)

we shall move the logic inside shouldDisableRename to ReportUtils and use it in RoomNamePage
https://github.com/Expensify/App/blob/6e5b07cf45944f28373e188d6f02620a12d6da29/src/pages/settings/Report/ReportSettingsPage.js#L63-L82
There are two approaches for this -

Wrap the component in FullPageNotFoundView and display the page not found view with error. Same check has been done in WriteCapabilityCheck, here we are not showing any error message.
If we want to show the error message, we pass it using subtitleKey.
add the check in RoomNamePage and if the room name can't be changed, we return and render nothing.

What alternative solutions did you explore? (Optional)

The bug is for announce, admin room name update which are not allowed by either admin or user. We are tricking the system here to update, not that 21498

I think if we already have a fix in the works, let's HOLD on that.

Held on https://github.com/Expensify/App/issues/21498

@twisterdotcom The fix was merged to staging couple of hours ago. We can re-test this issue on staging

Okay, this is fixed. I will pay out for the bug report here ONLY.
Paying @dhanashree-sawant $250 via Upwork: https://www.upwork.com/nx/wm/pre-hire/c/8577561/offer/25810014

Paid.


Easy Shopping Experience

Easy Shopping Experience
User Story
AS A user for the application
I WANT to have a good user experience
SO THAT I can meet my end goals for using the application
Acceptance Criteria
GIVEN the user would like to have a good user experience
WHEN they enter the application to shop
THEN they will easily find the menu and cart

This has been completed


Upstream non-Phase Change additions from #179

Upstream non-Phase Change additions from #179.

thanks for doing this @henryleberre -- this will be convenient.

@henryleberre, I need clarification on part of this PR. You added the test cases corresponding to phase change, but the phase change feature has not been added to MFC yet (the PR is still awaiting revision). Is this intentional? It seems more natural to add those test cases when the phase change features are added.

This is the case some overzealous grep'ing on my end - let me fix that now!

@sbryngelson I removed the phase-change tests.

@henryleberre they still appear to be there? There are also diffs to non-phase-change test case goldenfiles that seem erroneous.

@sbryngelson, apologies.. I have no idea what happened there. I just ran - again:
~/MFC-JRChreim $ git reset --soft HEAD~1
~/MFC-JRChreim $ git restore --staged $(git status | grep README | awk '{print $3}' | sed s/README\.md//g | tr '\n' ' ')
~/MFC-JRChreim $ git commit -m 'Internal energy calculation bugfix' --author="JRChreim<EMAIL_ADDRESS>
The tests whose golden files were modified are these:
~/MFC-JRChreim $ ./mfc.sh test -l | grep 'model_eqns=3'
  2F35A1FE   1D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
  FD891191   1D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3
  043B535A   2D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
  4F2F4ACE   2D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3
  B89B8C70   2D -> Axisymmetric -> model_eqns=3
  6241177B   3D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
  F0E6771E   3D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3

These were meant to be updated by @JRChreim, fixing the internal energy calculation for model_eqns=3.


Update FAQ in UG

Update FAQ in UG by adding more questions and answers
Add 2 quotes that signifies the start and end of a friendship. I feel that it adds human touch to our user guide. Let's ask Ms Serene about it later.

Interesting, I like this idea!


Service

Is there some idea for getting the users by a json file on a service and not locally?

As the users you are referring to user roles?
The ones that are here ?


Add requirements.txt to all quickstarts

Fixes #2.

Adds the standard requirements.txt file to each quickstart directory.

Rather than "pip install ", we are using the standard "pip install -r".
This change is independent of DevSite instructions.

This ensures the compatibility of the dependencies so that we don't see a quickstart randomly break.
Tested locally.
Ideally the user has venv set up, but we've never enforced that or added that to the quickstart docs.
This requirements.txt setup is similar to what we can see in the GCP python repo.

Assigning to @sqrrrl, since he is on-call this week.


How should I use the newest three.dart

I saw that on Dart Pub the three.dart package is pretty outdated [https://pub.dartlang.org/packages/three]. How should I used the newest three.dart in my project?
Thanks!

You can specify a git repository as a pub dependency.  It's not ideal, but
it's the only choice we have right now for three.dart.  See
https://www.dartlang.org/tools/pub/dependencies.html
On Wed, Mar 11, 2015 at 11:21 AM, Jiakui<EMAIL_ADDRESS>wrote:

I saw that on Dart Pub the three.dart package is pretty outdated [
https://pub.dartlang.org/packages/three]. How should I used the newest
three.dart in my project?
Thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/threeDart/three.dart/issues/190.

Furthermore, you can prevent merges into master from potentially breaking your code by forking the three.dart git repository and then listing your copy of the repository as the dependency in your pubspec.yaml. If you do it that way you can wait to pull merges from the main repository until you are ready to deal with any problems it causes in your projects that depend on three.dart.

Alternatively, you can peg a particular commit right in the pubspec.
On Wednesday, March 11, 2015, Christopher Grabowski <
<EMAIL_ADDRESS>wrote:

Furthermore, you can prevent merges into master from potentially breaking
your code by forking the three.dart git repository and then listing your
copy of the repository as the dependency in your pubspec.yaml. If you do it
that way you can wait to pull merges from the main repository until you are
ready to deal with any problems it causes in your projects that depend on
three.dart.
—
Reply to this email directly or view it on GitHub
https://github.com/threeDart/three.dart/issues/190#issuecomment-78312029
.

Oh cool! I didn't know that.


App-platform rework

As a company, we have now (spring 2023) agreed on 2 major objectives:

Enhance our "managed apps" offering into a full-fledged "internal developer portal"
Allow installation of our apps, especially the full "developer portal" tooling into non-giant-swarm customers: so customers that run and manage their clusters with 3rd party providers, like EKS or AKS, but still want to use our apps (like "security stack" or full "developer platform") offering. To make it possible, we will use our concept of Management Clusters (MCs) and App Platform to make it possible to offer the apps for a fleet of clusters.

To make these goals achievable, we, as team honey badger, see the following problems and challenges:

Complex setup of App Platforms on MCs

So far, we never let customers (even "customer admins") to have full access permissions on MCs, even though we allow them to install their own apps there (this feature has become very important for some customers). This means, that for any new app or any new feature we implement on MCs, we have to make sure that there's no escalation path that can be used by customers to escalate their permissions to full cluster level admin.
As a result of above, we run 2 flux deployments for each installation: one that handles customer's apps and one that handles giant swarm apps. This setup is complex, requires complex RBAC setup, makes flux management hard and RBAC management even more complex.

Very custom solution for app platform and app delivery.

To fulfill our needs, we use a combination of in-house projects, mainly app-operator and chart-operator to deliver applications and config and konfigure to deliver them.
Functionality of app-operator and chart-operator is now offered natively by very good tools like Flux (that wasn't the case when these tools were created).
Current setup of config+konfigure is no longer acceptable, as it forces us to share the same config repo across all the MCs and additionally forces us to maintain our fork of how flux is distributed, where we can inject our own configuration tools.

As we expect that enhancing the product in these directions will create more configuration and deployment cases, more problems with RBAC management, we want to start this with a set of simplification efforts, that will make the setup much easier to manager, handle more cases and maintain.
As such, we see the follwing as important simplification steps that will make enhancing the product scope easier:
### Tasks
- [ ] https://github.com/giantswarm/roadmap/issues/2455
- [ ] https://github.com/giantswarm/roadmap/issues/2456
- [ ] https://github.com/giantswarm/roadmap/issues/2457
- [ ] https://github.com/giantswarm/giantswarm/issues/26980
- [ ] https://github.com/giantswarm/roadmap/issues/2490
- [ ] https://github.com/giantswarm/giantswarm/issues/24784

@piontec this issue has been in watching for over a year now. Is there a way forward or can this issue be closed?

This is still in progress, but with low priority. Let's keep it as it is.


chore(spanner): update region tags

Description
See internal links
Update sample region tags to match other languages.

Please don't publish links to internal resources 😄


increase unit test coverage in controller

Current Behavior
5% unit test coverage
Proposed Behavior
60% unit test coverage
Solution Proposal

68% UT coverage rate


added preview button with link that narrows you to the stream

#3878

Automated message from Dropbox CLA bot
@speed-force, thanks for the pull request! It looks like you haven't yet signed the Dropbox CLA. Please sign it here.

Automated message from Dropbox CLA bot
@speed-force, thanks for signing the CLA!

@speed-force You have some failing front-end node tests (./tools/test-js-with-node) . Please fix it. You also have some lint errors (./tools/lint-all). You can run all the tests by running the command ./tools/-all.
When you add a new front end feature it's always a good idea to post the image.
Thanks for working on this :)

@speed-force Thanks for working on this. have posted some comments. You should also merge the two commits into one.  Rest looks great :)

Can we rename the button's class to something like preview-stream-button?  We have another "preview" feature for compose, and I want to avoid accidental name collisions.

This works, which is great, but the CSS needs more work:

The text should vertically align with that in the adjacent button
The buttons should have a reasonable space between them, like in bottom of the message pane with compose closed.
Also it's weird you've got so much !important in there.

Can you fix those @speed-force?
Also clean up your commit message to match our style;
Check out the Zulip commit message guidelines for more details: https://zulip.readthedocs.io/en/latest/version-control.html#commit-messages

Thanks for working on this, @speed-force!  I merged this as 5023b69ca5d867cdcffbcb58250d2c2b4d7a5733, and I amended your commit message a bit.


input pipeline spends all time in QueueDequeueMany

My training process use tfrecord format for train&eval dataset .
I test the benchmark of reader , only 8000records/second.  and io speed(see from iotop command) just 400KB-500KB/s.
I'm using the cpp version of protobuf here https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md#protobuf-library-related-issues
What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
https://github.com/tensorflow/tensorflow/issues/4467
https://github.com/tensorflow/tensorflow/issues/4732
If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
def read_and_decode(filename_queue):
     reader = tf.TFRecordReader()
    _, serialized_example = reader.read(filename_queue)
    return serialized_example

  serialized_example = read_and_decode(filename_queue)
  batch_serialized_example = tf.train.shuffle_batch(
      [serialized_example],
      batch_size=batch_size,
      num_threads=thread_number,
      capacity=capacity,
      min_after_dequeue=min_after_dequeue)
  features = tf.parse_example(
      batch_serialized_example,
      features={
          "label": tf.FixedLenFeature([], tf.float32),
          "ids": tf.VarLenFeature(tf.int64),
          "values": tf.VarLenFeature(tf.float32),
      })

What other attempted solutions have you tried?
I try to set num_threads in  tf.train.shuffle_batch but not working. It seems that when set to 2 threads, it work at 8000records/s, when enlarge the thread number, it get slower. (I remove all ops that cost cpus. Just read data.)
My sever are 24 core cpus.

here is my benchmark script and timeline result (timeline.json original file inlcude)
https://gist.github.com/ericyue/7705407a88e643f7ab380c6658f641e8

https://github.com/tensorflow/tensorflow/issues/4732
I read this issue that you guys talk about, seems the same problem. I'am using the latest version of tensorflow , but it's still unsolved.
@gunan  @ebrevdo  @yaroslavvb

It's not clear if it's a bug in TensorFlow or somewhere else....can you give a reproducible example? (ie, I can't run your example because it's missing the input files).
Also, the time Dequeue op takes is surprisingly similar to the time in https://github.com/tensorflow/tensorflow/issues/4740 . The issue there was queue starvation. Python thread scheduler always scheduled compute thread ahead of data prefetching thread, so at every step the compute thread was blocked in Dequeue, and the time to context switch, read single example and switch back to compute was about 10 ms.
Can you follow advice at the top of http://stackoverflow.com/a/39842628/419116 and see if that helps?

@yaroslavvb    I upload a part of my dataset here.
20161230_part-00390.output.pb.zlib.zip

increase queue size to fit the entire dataset
the "queue size " you mentioned is capacity in tf.train.shuffle_batch() ?
in my script, capacity = thread_number * batch_size + min_after_dequeue, I tried increase min_after_dequeue or batchsize to a large value but it didn't help.
print(sess.run(queue.size()))   how to get queue variable in my script?

You could just add time.sleep(30) after starting queue runners and checking
if it makes things faster. If not and get timeline for one of the first few
iterations
On Jan 14, 2017 8:27 PM, "Eric Yue"<EMAIL_ADDRESS>wrote:

@yaroslavvb https://github.com/yaroslavvb I upload a part of my dataset
here.
20161230_part-00390.output.pb.zlib.zip
https://github.com/tensorflow/tensorflow/files/706423/20161230_part-00390.output.pb.zlib.zip

increase queue size to fit the entire dataset
the "queue size " you mentioned is capacity in tf.train.shuffle_batch()
?
in my script, capacity = thread_number * batch_size + min_after_dequeue,
I tried increase min_after_dequeue or batchsize to a large value but it
didn't help.
print(sess.run(queue.size())) how to get queue variable in my
script?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/6845#issuecomment-272672950,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABaHJ2DAlp948wIIJ3YozsdPQPlaBTaks5rSaAYgaJpZM4Ljje7
.

@yaroslavvb   I do as you said.
After I add time.sleep and increase capacity in tf.batch, the speed changed from 7k/s to 70k/s.
But I find two thinks :

the speed become slower with the time elapse.
then increase the thread number , the speed become slow.  the best number is  2.(my server are 24cores cpu)

and if the reason why it's slow is queue starvation, how to solve this? manually time.sleep every batch? is there any better solution ?

I have read the post you answered http://stackoverflow.com/questions/39840323/benchmark-of-howto-reading-data/39842628#39842628
you said
The solution is to increase number of Python threads, increase queue size to fit the entire dataset, start queue runners, and then pause main thread for a couple of seconds to give queue runners to pre-populate the queue.

so my question is :
1、 The solution is to increase number of Python threads
do you mean increase the thread number in tf.train.shuffle_batch() ? I tried it . but not working . increase the number of thread can descrease the speed.
2、 increase queue size to fit the entire dataset
increase the capacity in tf.train.shuffle_batch() ?  It seems that I didn't use queue directly, so i confused that what is the queue size you mean.
3、pause main thread for a couple of seconds
this maybe work, but only a few second later it's slow again.

BTW in iotop command , I can only see one tensorflow thread reading disk io when I set thread number in tf.shufle_batch . (sometime show 2 threads. )

I suppose this is a "bug" in how Python works. What happens when main compute thread session.run call is too fast (ie, <2ms), then Python does not switch to the other reading read and will keep running your compute thread until it runs out of data to consume. At which point it'll switch to "read one example-run one train step-read another example", which is much less efficient than reading many examples before switching threads to do computation on them.
This is not a common scenario because usually: train_op is slow/compute-intensive, and the pipeline is bottlenecked by compute part rather than IO part.
If adding time.sleep helps at first, and then training slows down again, suggests that enqueing part of the pipeline has lower throughput than your dequeing half.
It could mean that your "dequeing" (ie training part) is fundamentally easy, and decoding can't keep up. It could mean that decoding examples is inefficient and could be sped up. So you could try benchmarking just the decoding part (ie, start queue runners, and see how fast they can add things to queue), and try to estimate if the speed at which this happens is close to maximum attainable

sorry I can not understand. You means the main reason it slows is because the dequeing half too fast?  In my acutal script , the training process is a 4-layer network each with 128 hidden unit. I think it's complex enough .
I'm confused. Increase the thread number can not speed up , so it maybe that the queue is always full, and the dequeing part is slow ?  it's not match with your thought about dequeing is too fase.
is there any other solution can I use to speed up ?

I'll try running your example on Monday, but meanwhile you could try benchmarking just the enqueuing part of the pipeline (no training, just adding data to queue) and seeing how fast it is.

thanks.  The script I provide is just without training. it's just read from file and get from queue.
I didn't know how to measure the time only adding data because I cannot get the timestamp when the queue is full. I'm not using the Queue directly.

You can call queue size operation directly from graph, ie
print(tf.get_default_graph().as_graph_def())
...
node {
name: "batch/fifo_queue_Size"
op: "QueueSize"
input: "batch/fifo_queue"
...
then you can do sess.run("batch/fifo_queue_Size") to get number of elements in that queue

there's a "shuffle_batch/random_shuffle_queue_Size", but value keeps None.
node {
  name: "matching_filenames/MatchingFiles/pattern"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "train/*/*.zlib"
      }
    }
  }
}
node {
  name: "matching_filenames/MatchingFiles"
  op: "MatchingFiles"
  input: "matching_filenames/MatchingFiles/pattern"
}
node {
  name: "matching_filenames"
  op: "Variable"
  attr {
    key: "container"
    value {
      s: ""
    }
  }
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "shape"
    value {
      shape {
        unknown_rank: true
      }
    }
  }
  attr {
    key: "shared_name"
    value {
      s: ""
    }
  }
}
node {
  name: "matching_filenames/Assign"
  op: "Assign"
  input: "matching_filenames"
  input: "matching_filenames/MatchingFiles"
  attr {
    key: "T"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@matching_filenames"
      }
    }
  }
  attr {
    key: "use_locking"
    value {
      b: true
    }
  }
  attr {
    key: "validate_shape"
    value {
      b: false
    }
  }
}
node {
  name: "matching_filenames/read"
  op: "Identity"
  input: "matching_filenames"
  attr {
    key: "T"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@matching_filenames"
      }
    }
  }
}
node {
  name: "input_producer/Size"
  op: "Size"
  input: "matching_filenames/read"
  attr {
    key: "T"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "out_type"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "input_producer/Greater/y"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
        }
        int_val: 0
      }
    }
  }
}
node {
  name: "input_producer/Greater"
  op: "Greater"
  input: "input_producer/Size"
  input: "input_producer/Greater/y"
  attr {
    key: "T"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "input_producer/Assert/Const"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "string_input_producer requires a non-null input tensor"
      }
    }
  }
}
node {
  name: "input_producer/Assert/Assert/data_0"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "string_input_producer requires a non-null input tensor"
      }
    }
  }
}
node {
  name: "input_producer/Assert/Assert"
  op: "Assert"
  input: "input_producer/Greater"
  input: "input_producer/Assert/Assert/data_0"
  attr {
    key: "T"
    value {
      list {
        type: DT_STRING
      }
    }
  }
  attr {
    key: "summarize"
    value {
      i: 3
    }
  }
}
node {
  name: "input_producer/Identity"
  op: "Identity"
  input: "matching_filenames/read"
  input: "^input_producer/Assert/Assert"
  attr {
    key: "T"
    value {
      type: DT_STRING
    }
  }
}
node {
  name: "input_producer/RandomShuffle"
  op: "RandomShuffle"
  input: "input_producer/Identity"
  attr {
    key: "T"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "seed"
    value {
      i: 87654321
    }
  }
  attr {
    key: "seed2"
    value {
      i:<PHONE_NUMBER>
    }
  }
}
node {
  name: "input_producer/limit_epochs/Const"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT64
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT64
        tensor_shape {
        }
        int64_val: 0
      }
    }
  }
}
node {
  name: "input_producer/limit_epochs/epochs"
  op: "Variable"
  attr {
    key: "container"
    value {
      s: ""
    }
  }
  attr {
    key: "dtype"
    value {
      type: DT_INT64
    }
  }
  attr {
    key: "shape"
    value {
      shape {
      }
    }
  }
  attr {
    key: "shared_name"
    value {
      s: ""
    }
  }
}
node {
  name: "input_producer/limit_epochs/epochs/Assign"
  op: "Assign"
  input: "input_producer/limit_epochs/epochs"
  input: "input_producer/limit_epochs/Const"
  attr {
    key: "T"
    value {
      type: DT_INT64
    }
  }
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@input_producer/limit_epochs/epochs"
      }
    }
  }
  attr {
    key: "use_locking"
    value {
      b: true
    }
  }
  attr {
    key: "validate_shape"
    value {
      b: true
    }
  }
}
node {
  name: "input_producer/limit_epochs/epochs/read"
  op: "Identity"
  input: "input_producer/limit_epochs/epochs"
  attr {
    key: "T"
    value {
      type: DT_INT64
    }
  }
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@input_producer/limit_epochs/epochs"
      }
    }
  }
}
node {
  name: "input_producer/limit_epochs/CountUpTo"
  op: "CountUpTo"
  input: "input_producer/limit_epochs/epochs"
  attr {
    key: "T"
    value {
      type: DT_INT64
    }
  }
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@input_producer/limit_epochs/epochs"
      }
    }
  }
  attr {
    key: "limit"
    value {
      i: 500
    }
  }
}
node {
  name: "input_producer/limit_epochs"
  op: "Identity"
  input: "input_producer/RandomShuffle"
  input: "^input_producer/limit_epochs/CountUpTo"
  attr {
    key: "T"
    value {
      type: DT_STRING
    }
  }
}
node {
  name: "input_producer"
  op: "FIFOQueue"
  attr {
    key: "capacity"
    value {
      i: 32
    }
  }
  attr {
    key: "component_types"
    value {
      list {
        type: DT_STRING
      }
    }
  }
  attr {
    key: "container"
    value {
      s: ""
    }
  }
  attr {
    key: "shapes"
    value {
      list {
        shape {
        }
      }
    }
  }
  attr {
    key: "shared_name"
    value {
      s: ""
    }
  }
}
node {
  name: "input_producer/input_producer_EnqueueMany"
  op: "QueueEnqueueMany"
  input: "input_producer"
  input: "input_producer/limit_epochs"
  attr {
    key: "Tcomponents"
    value {
      list {
        type: DT_STRING
      }
    }
  }
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@input_producer"
      }
    }
  }
  attr {
    key: "timeout_ms"
    value {
      i: -1
    }
  }
}
node {
  name: "input_producer/input_producer_Close"
  op: "QueueClose"
  input: "input_producer"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@input_producer"
      }
    }
  }
  attr {
    key: "cancel_pending_enqueues"
    value {
      b: false
    }
  }
}
node {
  name: "input_producer/input_producer_Close_1"
  op: "QueueClose"
  input: "input_producer"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@input_producer"
      }
    }
  }
  attr {
    key: "cancel_pending_enqueues"
    value {
      b: true
    }
  }
}
node {
  name: "input_producer/input_producer_Size"
  op: "QueueSize"
  input: "input_producer"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@input_producer"
      }
    }
  }
}
node {
  name: "input_producer/Cast"
  op: "Cast"
  input: "input_producer/input_producer_Size"
  attr {
    key: "DstT"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "SrcT"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "input_producer/mul/y"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_FLOAT
        tensor_shape {
        }
        float_val: 0.03125
      }
    }
  }
}
node {
  name: "input_producer/mul"
  op: "Mul"
  input: "input_producer/Cast"
  input: "input_producer/mul/y"
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
}
node {
  name: "input_producer/queue/input_producer/fraction_of_32_full/tags"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "input_producer/queue/input_producer/fraction_of_32_full"
      }
    }
  }
}
node {
  name: "input_producer/queue/input_producer/fraction_of_32_full"
  op: "ScalarSummary"
  input: "input_producer/queue/input_producer/fraction_of_32_full/tags"
  input: "input_producer/mul"
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
}
node {
  name: "TFRecordReader"
  op: "TFRecordReader"
  attr {
    key: "compression_type"
    value {
      s: "ZLIB"
    }
  }
  attr {
    key: "container"
    value {
      s: ""
    }
  }
  attr {
    key: "shared_name"
    value {
      s: ""
    }
  }
}
node {
  name: "ReaderRead"
  op: "ReaderRead"
  input: "TFRecordReader"
  input: "input_producer"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@TFRecordReader"
        s: "loc:@input_producer"
      }
    }
  }
}
node {
  name: "shuffle_batch/random_shuffle_queue"
  op: "RandomShuffleQueue"
  attr {
    key: "capacity"
    value {
      i: 50000
    }
  }
  attr {
    key: "component_types"
    value {
      list {
        type: DT_STRING
      }
    }
  }
  attr {
    key: "container"
    value {
      s: ""
    }
  }
  attr {
    key: "min_after_dequeue"
    value {
      i: 1100
    }
  }
  attr {
    key: "seed"
    value {
      i: 0
    }
  }
  attr {
    key: "seed2"
    value {
      i: 0
    }
  }
  attr {
    key: "shapes"
    value {
      list {
        shape {
        }
      }
    }
  }
  attr {
    key: "shared_name"
    value {
      s: ""
    }
  }
}
node {
  name: "shuffle_batch/random_shuffle_queue_enqueue"
  op: "QueueEnqueue"
  input: "shuffle_batch/random_shuffle_queue"
  input: "ReaderRead:1"
  attr {
    key: "Tcomponents"
    value {
      list {
        type: DT_STRING
      }
    }
  }
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@shuffle_batch/random_shuffle_queue"
      }
    }
  }
  attr {
    key: "timeout_ms"
    value {
      i: -1
    }
  }
}
node {
  name: "shuffle_batch/random_shuffle_queue_Close"
  op: "QueueClose"
  input: "shuffle_batch/random_shuffle_queue"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@shuffle_batch/random_shuffle_queue"
      }
    }
  }
  attr {
    key: "cancel_pending_enqueues"
    value {
      b: false
    }
  }
}
node {
  name: "shuffle_batch/random_shuffle_queue_Close_1"
  op: "QueueClose"
  input: "shuffle_batch/random_shuffle_queue"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@shuffle_batch/random_shuffle_queue"
      }
    }
  }
  attr {
    key: "cancel_pending_enqueues"
    value {
      b: true
    }
  }
}
node {
  name: "shuffle_batch/random_shuffle_queue_Size"
  op: "QueueSize"
  input: "shuffle_batch/random_shuffle_queue"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@shuffle_batch/random_shuffle_queue"
      }
    }
  }
}
node {
  name: "shuffle_batch/sub/y"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
        }
        int_val: 1100
      }
    }
  }
}
node {
  name: "shuffle_batch/sub"
  op: "Sub"
  input: "shuffle_batch/random_shuffle_queue_Size"
  input: "shuffle_batch/sub/y"
  attr {
    key: "T"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "shuffle_batch/Maximum/x"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
        }
        int_val: 0
      }
    }
  }
}
node {
  name: "shuffle_batch/Maximum"
  op: "Maximum"
  input: "shuffle_batch/Maximum/x"
  input: "shuffle_batch/sub"
  attr {
    key: "T"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "shuffle_batch/Cast"
  op: "Cast"
  input: "shuffle_batch/Maximum"
  attr {
    key: "DstT"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "SrcT"
    value {
      type: DT_INT32
    }
  }
}
node {
  name: "shuffle_batch/mul/y"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_FLOAT
        tensor_shape {
        }
        float_val: 2.04498974199e-05
      }
    }
  }
}
node {
  name: "shuffle_batch/mul"
  op: "Mul"
  input: "shuffle_batch/Cast"
  input: "shuffle_batch/mul/y"
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
}
node {
  name: "shuffle_batch/queue/shuffle_batch/fraction_over_1100_of_48900_full/tags"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "shuffle_batch/queue/shuffle_batch/fraction_over_1100_of_48900_full"
      }
    }
  }
}
node {
  name: "shuffle_batch/queue/shuffle_batch/fraction_over_1100_of_48900_full"
  op: "ScalarSummary"
  input: "shuffle_batch/queue/shuffle_batch/fraction_over_1100_of_48900_full/tags"
  input: "shuffle_batch/mul"
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
}
node {
  name: "shuffle_batch/n"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
        }
        int_val: 5000
      }
    }
  }
}
node {
  name: "shuffle_batch"
  op: "QueueDequeueMany"
  input: "shuffle_batch/random_shuffle_queue"
  input: "shuffle_batch/n"
  attr {
    key: "_class"
    value {
      list {
        s: "loc:@shuffle_batch/random_shuffle_queue"
      }
    }
  }
  attr {
    key: "component_types"
    value {
      list {
        type: DT_STRING
      }
    }
  }
  attr {
    key: "timeout_ms"
    value {
      i: -1
    }
  }
}
node {
  name: "ParseExample/Const"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_FLOAT
        tensor_shape {
          dim {
          }
        }
      }
    }
  }
}
node {
  name: "ParseExample/ParseExample/names"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
          dim {
          }
        }
      }
    }
  }
}
node {
  name: "ParseExample/ParseExample/sparse_keys_0"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "ids"
      }
    }
  }
}
node {
  name: "ParseExample/ParseExample/sparse_keys_1"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "values"
      }
    }
  }
}
node {
  name: "ParseExample/ParseExample/dense_keys_0"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_STRING
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_STRING
        tensor_shape {
        }
        string_val: "label"
      }
    }
  }
}
node {
  name: "ParseExample/ParseExample"
  op: "ParseExample"
  input: "shuffle_batch"
  input: "ParseExample/ParseExample/names"
  input: "ParseExample/ParseExample/sparse_keys_0"
  input: "ParseExample/ParseExample/sparse_keys_1"
  input: "ParseExample/ParseExample/dense_keys_0"
  input: "ParseExample/Const"
  attr {
    key: "Ndense"
    value {
      i: 1
    }
  }
  attr {
    key: "Nsparse"
    value {
      i: 2
    }
  }
  attr {
    key: "Tdense"
    value {
      list {
        type: DT_FLOAT
      }
    }
  }
  attr {
    key: "dense_shapes"
    value {
      list {
        shape {
        }
      }
    }
  }
  attr {
    key: "sparse_types"
    value {
      list {
        type: DT_INT64
        type: DT_FLOAT
      }
    }
  }
}
node {
  name: "init"
  op: "NoOp"
  input: "^matching_filenames/Assign"
}
node {
  name: "init_1"
  op: "NoOp"
  input: "^input_producer/limit_epochs/epochs/Assign"
}
versions {
  producer: 17
}

Correction, sess.run("batch/fifo_queue_Size:0"). Also getting timeline for actual enqueue step will tell where true slowness is, maybe also combined with CPU profile from Google perf tools

BTW, when I run your example on MacBook everything seems to be running consistently between 13k and 17k examples per second
[2017-01-15 21:03:09] time[  0.12] step[        20] speed[ 17000]
[2017-01-15 21:03:09] time[  0.14] step[        40] speed[ 14747]
[2017-01-15 21:03:10] time[  0.14] step[        60] speed[ 14788]
[2017-01-15 21:03:10] time[  0.14] step[        80] speed[ 14053]
[2017-01-15 21:03:10] time[  0.14] step[       100] speed[ 14386]
[2017-01-15 21:03:10] time[  0.13] step[       120] speed[ 14964]
[2017-01-15 21:03:10] time[  0.14] step[       140] speed[ 14594]
[2017-01-15 21:03:10] time[  0.13] step[       160] speed[ 15027]
[2017-01-15 21:03:10] time[  0.14] step[       180] speed[ 14646]

I try on my macbook pro too (MacBook Pro (Retina, 13-inch, Early 2015))
I'm wondering why the different come from.
I'm using tensorflow 0.12.1 and install from pip. And using the cpp-implemention protobuf in python.
[2017-01-15 21:45:32] time[  0.26] step[        20] speed[  7772]
[2017-01-15 21:45:32] time[  0.38] step[        40] speed[  5326]
[2017-01-15 21:45:33] time[  0.50] step[        60] speed[  3962]
[2017-01-15 21:45:33] time[  0.37] step[        80] speed[  5388]
[2017-01-15 21:45:34] time[  0.54] step[       100] speed[  3703]
[2017-01-15 21:45:34] time[  0.62] step[       120] speed[  3251]
[2017-01-15 21:45:35] time[  0.34] step[       140] speed[  5893]
[2017-01-15 21:45:35] time[  0.27] step[       160] speed[  7454]
[2017-01-15 21:45:35] time[  0.27] step[       180] speed[  7323]
[2017-01-15 21:45:36] time[  0.48] step[       200] speed[  4145]
[2017-01-15 21:45:36] time[  0.34] step[       220] speed[  5958]
[2017-01-15 21:45:37] time[  0.42] step[       240] speed[  4734]
[2017-01-15 21:45:37] time[  0.49] step[       260] speed[  4100]

@yaroslavvb   hi, I tried as you said to try benchmarking just the enqueuing part of the pipeline, the result is 7058records/s on my macbook.
I'm using a large min_after_dequeue to ensure 5 seconds can not load all the records.
    num1 = sess.run("shuffle_batch/random_shuffle_queue_Size:0")
    start_time = datetime.datetime.now(pytz.timezone('Asia/Shanghai'))
    threads = tf.train.start_queue_runners(coord=coord, sess=sess)
    time.sleep(5)
    end_time = datetime.datetime.now(pytz.timezone('Asia/Shanghai'))
    sec = (end_time - start_time).total_seconds()
    num2 = sess.run("shuffle_batch/random_shuffle_queue_Size:0")
    speed = (num2-num1)/sec

I tried on my server , the speed is 7567/s .
So it seems that the bottleneck  is read speed ?   Why increase thread number can speed up ?

@yaroslavvb You've offered some really excellent guidance in this thread. The problem appears to be resolved from my perspective. I'm going to close this out since I'm not seeing evidence of a bug in TensorFlow. But by all means, feel free to continue discussing.

I'm sorry but where the solution above ?
The every solution in stackoverflow or older issue is not possible.
I do speed up by add a time.sleep in the beginning , but a seconds later it slower again. So this is the solution ?
pool performance is not a bug ?

I didn't know why increase thread number can not speed up  enqueuing speed .
can you try to use my script on your macbook ?
just try thread number 1 and 5.  I do this on 3 server and 2 macbooks.
The result shows that increase thread do decrease the speed.
@yaroslavvb

@yaroslavvb was unable to reproduce the performance decline you describe. The problem seems related to how TensorFlow is being used w.r.t. Python scheduling rather than a bug in TensorFlow itself. As for increasing threads decreasing speed, that's because GIL causes Python threads to have negatively scalability. For many types of computations, particularly CPU-bound ones, 1 + 1 < 1.

Thanks for reply.
He reproduced speed at 13k-17k , but I think if he add time.sleep in the main thread it may speed up to 60k or higher.
So do you think the enqueuing speed at 13k-17k is a reasonable speed?  If so I will accept it .
And I am wondering that if it is a python-gil issue , Why tensorflow support multi-thread or something to support a useless (it seems slower than single thread) function ?

ok, forget it. even though can not solved the problem, but I do realized more about the tf.
I will try to dig more and see if I can find the reason.
@yaroslavvb  @jart   Thanks for your help :)

The TensorFlow team is glad to help. If you manage to pinpoint the root cause of a performance issue, and are able to help us understand what specific part needs to be improved, please report back and we'll be more than happy to listen.

@ericyue this issue is that you are issuing too many session.run calls. With dequeue_many option to batch you can get 10x speed-up, put analysis [here](http://stackoverflow.com/questions/41647784/tfrecordreader-seems-extremely-slow-and-multi-threads-reading-not-working/41688401#41688401

@mrry -- this is an interesting use-case to keep in mind for your input pipelines work. To summarize, with current situation, if you have an input pipeline that produces >10k items/second in an intermediate stage, that's going to cause 10k session.run(enqueue_op) calls/second which is too much for Python to handle. An automatic solution would batch this up into fewer enqueue_many calls

Sorry to comment on a closed thread. I believe I'm running into a similar issue since I have a relatively simple feed-forward network with lots of examples that I want to feed through it. I've generated a trace file which shows my process dominated by QueueDequeueMany, and the GPUs oscillate between 0% and 30-60% usage. I have tried different time.sleep settings to give the queue time to fill, increased the number of preprocess threads, and increased the batch size, but none have had a noticeable effect.
I am trying the tf.train.batch(..., enqueue_many=True) approach outlined above, but I get errors when I start using multiple GPUs.

E tensorflow/core/util/events_writer.cc:62] Could not open events file: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785002.server: Resource exhausted: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785002.server
E tensorflow/core/util/events_writer.cc:95] Write failed because file could not be opened.
E tensorflow/core/util/events_writer.cc:62] Could not open events file: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785004.server: Resource exhausted: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785004.server
E tensorflow/core/util/events_writer.cc:95] Write failed because file could not be opened.

I do not get this error if I set num_gpus=1. I need multiple GPUs because speed is an issue due to the quantity of data I am feeding through it.
Is there something I need to do differently to use enqueue_many=True with multiple GPUs?

A quick google search shows "Resource Exhausted" on writing event file can
be caused by running out of memory
http://stackoverflow.com/questions/38340070/after-500-steps-tensorflow-fail-to-write-summaries
On Wed, Jan 18, 2017 at 4:38 PM, Erik @ MileIQ<EMAIL_ADDRESS>wrote:

Sorry to comment on a closed thread. I believe I'm running into a similar
issue since I have a relatively simple feed-forward network with lots of
examples that I want to feed through it. I've generated a trace file which
shows my process dominated by QueueDequeueMany, and the GPUs oscillate
between 0% and 30-60% usage. I have tried different time.sleep settings
to give the queue time to fill, increased the number of preprocess threads,
and increased the batch size, but none have had a noticeable effect.
I am trying the tf.train.batch(..., enqueue_many=True) approach outlined
above, but I get errors when I start using multiple GPUs.
E tensorflow/core/util/events_writer.cc:62] Could not open events file: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785002.server: Resource exhausted: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785002.server
E tensorflow/core/util/events_writer.cc:95] Write failed because file could not be opened.
E tensorflow/core/util/events_writer.cc:62] Could not open events file: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785004.server: Resource exhausted: /data_full/output/full/checkpoint-events/train/events.out.tfevents.1484785004.server
E tensorflow/core/util/events_writer.cc:95] Write failed because file could not be opened.
I do not get this error if I set num_gpus=1. I need multiple GPUs because
speed is an issue due to the quantity of data I am feeding through it.
Is there something I need to do differently to use enqueue_many=True with
multiple GPUs?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/6845#issuecomment-273647478,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABaHDegEBsiCB9GBEySZEzo5vmC3LdMks5rTrCbgaJpZM4Ljje7
.

Thanks for the quick response @yaroslavvb.
I just now discovered tf.ReaderBase.read_up_to, which seems to solve the problem without the extra code of queue_batch = [] ... queue_batch.append(serialized) ... tf.train.batch(..., enqueue_many=True). Now reading is able to keep up with the training and the queue grows over time instead of quickly depleting.

For posterity:
I believe it was a result of a race condition, since it was resolved by adding a sleep command before creating a tf.summary.FileWriter object.
Stepping back, the reason why I believe I have the same issue as described here: when I increase the batch size and time for the queue to fill up, I get much faster speeds when the queue is full (~35k examples/gpu/sec). Once the queue depletes, it falls back to (~5k examples/gpu/sec), which is about what I get with much smaller batch sizes. I also get smaller queue sizes when I increase the number of threads. This leads me to believe that the input is the bottleneck for training.
Here is the trace at step 1 when the queue is not depleted. UnsortedSegmentSum on the CPU dominates, I think due to the summing of the variables across GPUs.

By step 200, the queue has been depleted and now the process is dominated by QueueDequeueMany:

Glad to see you were able to find a solution.
Your resource exhausted error may have been due to opening too many files at once and running out of file descriptors. Supervisor has multiple threads doing things concurrently and can cause such issues due to unlucky timing. This is one problem that MonitoredSession/MonitoredTrainingSession were designed to solve -- everything is done in one thread there.
Long time spent in Dequeue is caused by Enqueue part being too slow. If you add time.sleep this masks the symptom, but doesn't solve underlying problem.

@yaroslavvb  wow! enqueue_many=True works great on my training process.  But I got resource  unavailabe  sometimes.  I'll try MonitoredSession/MonitoredTrainingSession  , it seems designd for  distribution training?

what's the difference between MonitoredTrainingSession and MonitoredSession  ? can not find the docs about these two .

I wonder if resource unavailable in your case is caused by too much memory. You can try disabling checkpoint writing and summary writing services to see if that helps. MonitoredTrainingSession and MonitoredSession are documented in comments right now in implementation file, main difference is that MonitoredTrainingSession is for distributed training with chief worker.