Material ModelViewSet DetailModelView: Limit the model fields shown in the details page

It seems that the DetailModelView currently does not  have to provision to hide fields. It always shows the all the fields of a model in the detail page.
We can use get_object_data method to override the fields to be displayed. But this doesn't seem to be a straightforward way.
Please advise if I am missing something.
Django-material version: 1.5.6

The best way to override object representation in the detail view is to override the view template http://docs.viewflow.io/frontend_crud.html#material.frontend.views.DetailModelView.get_template_names


STree matches on non-existent matches in edge cases

I came across some spurious matches on my data. I've whittled down the text to reproduce, so here's a minimal example:
from suffix_trees.STree import STree
text = "name language w en url http w namelanguage en url http"
stree = STree(text)

print "STree finds 'law' at index:", stree.find_all('law')
print "'law' in text -->", 'law' in text

which outputs
STree finds 'law' at index: [5]
'law' in text --> False

Thanks!
Fixed the bug.


Using parameter "all" for service "ask_for_update" gives error in v0.5.4

Running v0.5.4 using
service: duofern.ask_for_update
data:
  all: true

gives this in the UI: Error calling service duofern.ask_for_update. Unknown error
and this in the HA log:
2023-04-02 18:38:51.208 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: local variable 'sets' referenced before assignment
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 451, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 684, in _async_call_service_step
    await service_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 1745, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1788, in _execute_service
    await self._hass.async_add_executor_job(
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/duofern/__init__.py", line 152, in ask_for_update
    getDuofernStick(hass).command(device_id, 'getStatus')
  File "/usr/local/lib/python3.10/site-packages/pyduofern/duofern_stick.py", line 576, in command
    self.duofern_parser.set(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyduofern/duofern.py", line 770, in set
    logger.debug(sets.keys())  # join(" ", sort keys sets)
UnboundLocalError: local variable 'sets' referenced before assignment
2023-04-02 18:38:51.215 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [547173666864] Error handling message: Unknown error (unknown_error) from xxx.xxx.xxx.xxx (Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 27, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 646, in handle_execute_script
    await script_obj.async_run(msg.get("variables"), context=context)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1524, in async_run
    await asyncio.shield(run.async_run())
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 409, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 453, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 476, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 451, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 684, in _async_call_service_step
    await service_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 1745, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1788, in _execute_service
    await self._hass.async_add_executor_job(
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/duofern/__init__.py", line 152, in ask_for_update
    getDuofernStick(hass).command(device_id, 'getStatus')
  File "/usr/local/lib/python3.10/site-packages/pyduofern/duofern_stick.py", line 576, in command
    self.duofern_parser.set(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyduofern/duofern.py", line 770, in set
    logger.debug(sets.keys())  # join(" ", sort keys sets)
UnboundLocalError: local variable 'sets' referenced before assignment

Interesting part:
  File "/config/custom_components/duofern/__init__.py", line 152, in ask_for_update
    getDuofernStick(hass).command(device_id, 'getStatus')
  File "/usr/local/lib/python3.10/site-packages/pyduofern/duofern_stick.py", line 576, in command
    self.duofern_parser.set(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pyduofern/duofern.py", line 770, in set
    logger.debug(sets.keys())  # join(" ", sort keys sets)
UnboundLocalError: local variable 'sets' referenced before assignment

Additional notes after a bit more of blackbox testing (to find a work around for updating all devices):
❓ Gives no error but likely does nothing (?):
service: duofern.ask_for_update
data: {}

❌ Gives an error (wrong syntax)
service: duofern.ask_for_update
data:
  device_id: all

❓ Gives no error and makes use of HA magic "all", but likely does nothing (according to debug logs which contain nothing):
service: duofern.ask_for_update
data:
  device_id:
    - all

❓ Gives no error and uses a group of devices (cover group), but likely does nothing (according to debug logs which contain nothing):
service: duofern.ask_for_update
data:
  device_id:
    - cover.all_covers

Before I update my automations.yaml and scripts.yaml the 7th time during the last weeks I want to be sure to have a working service (syntax) 😃

Regarding the original error

File "/usr/local/lib/python3.10/site-packages/pyduofern/duofern.py", line 770, in set
logger.debug(sets.keys())  # join(" ", sort keys sets)
UnboundLocalError: local variable 'sets' referenced before assignment

You mentioned that you had bogous devices (ones that you didn't pair yourself) detected by duofern - I believe having a potentially non-existing device type triggered this. Sets were always set for "existing" duofern device types - therefore this didn't show up before. but the bogous devices seem to not match that description. I fixed duofern to not raise that error any more (will be in 0.5.4). you will likely be able to trigger this by running the update exclusively by updating one of the "bogous" devices.
Regarding "how to use all"
Your initial attempt is fine.
service: duofern.ask_for_update
data:
  all: true

Is the yaml the UI also produces when checking the box and it works on my side. The error it triggered on your side because of the ghost devices should be fixed with 0.5.4. (can't test for lack of ghost devices).
Regarding the logging:
Already in 0.5.3 there should always be info level logs when running the ask_for_update. They look like this:
2023-04-02 21:19:27.503 INFO (SyncWorker_7) [custom_components.duofern] asking ffffff for update

If they are not there the command didn't go out. This may have been caused by the duofern thread crashing due to the issue you describe above.
Nevertheless I added a bunch more verbosity to the update function - it might help in case we get another issue with bogous devices.

Interesting. I used the service pointing at only one specific device_id with debug logging enabled and there is - no - log entry.
Will update to 0.5.5.

Now with 0.5.5 I get this:
When running
1. As intended
service: duofern.ask_for_update
data:
  all: true

--> Absolutely NOTHING in debug log.
2. As not intended
service: duofern.ask_for_update
data: {}

--> gives 2023-04-02 23:58:02.891 WARNING (SyncWorker_0) [custom_components.duofern] Found no valid device ids???
❓

My fault. I used the "enable debug logging" from the GUI:

...which created no logs for duofern at all.
But using
service: logger.set_level
data:
  custom_components.duofern: debug

and firing
service: duofern.ask_for_update
data:
  all: true

afterwards gave the expected log output. (I used the same service to set it back to "warning" for duofern) Unfortunately it reminded me of my plenty ghost devices, I think I need a separate issue for them :-(
So I can confirm the new "all" syntax is working. I also did not see this issue anymore since on 0.5.5. So I closed this one.


java.lang.NullPointerException: text

I've been testing the conversion of some exported FOXML with this command:
java -jar migration-utils-6.1.0-driver.jar
  --source-type exported \
  --exported-dir archive \
  --target-dir ocfl
  --debug

The majority of objects are processed fine, but it has been running into this error occasionally (20/1000 objects):
DEBUG 14:03:08.394 (UserProvidedPidListManager) PID: druid:bb059dp5973, accept? true
DEBUG 14:03:08.394 (ResumePidListManager) PID: druid:bb059dp5973, accept? true
INFO 14:03:08.394 (Migrator) Processing "druid:bb059dp5973"...
DEBUG 14:03:08.408 (ArchiveGroupHandler) Committing object <info:fedora/druid:bb059dp5973>
ERROR 14:03:08.424 (Migrator) MIGRATION_FAILURE: pid="druid:bb059dp5973", message="text"
java.lang.NullPointerException: text
        at java.base/java.util.Objects.requireNonNull(Objects.java:233)
        at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1955)
        at java.base/java.time.Instant.parse(Instant.java:399)
        at org.fcrepo.migration.handlers.ocfl.ArchiveGroupHandler.createDatastreamHeaders(ArchiveGroupHandler.java:618)
        at org.fcrepo.migration.handlers.ocfl.ArchiveGroupHandler.processObjectVersions(ArchiveGroupHandler.java:256)
        at org.fcrepo.migration.handlers.ObjectAbstractionStreamingFedoraObjectHandler.completeObject(ObjectAbstractionStreamingFedoraObjectHandler.java:84)
        at org.fcrepo.migration.foxml.FoxmlInputStreamFedoraObjectProcessor.complete(FoxmlInputStreamFedoraObjectProcessor.java:223)
        at org.fcrepo.migration.foxml.FoxmlInputStreamFedoraObjectProcessor.lambda$processObject$0(FoxmlInputStreamFedoraObjectProcessor.java:218)
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
        at org.fcrepo.migration.foxml.FoxmlInputStreamFedoraObjectProcessor.processObject(FoxmlInputStreamFedoraObjectProcessor.java:218)
        at org.fcrepo.migration.Migrator.run(Migrator.java:161)
        at org.fcrepo.migration.PicocliMigrator.call(PicocliMigrator.java:328)
        at org.fcrepo.migration.PicocliMigrator.call(PicocliMigrator.java:51)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
        at picocli.CommandLine.access$900(CommandLine.java:145)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2101)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2068)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
        at picocli.CommandLine.execute(CommandLine.java:1864)
        at org.fcrepo.migration.PicocliMigrator.main(PicocliMigrator.java:175)

The resulting object does not appear to be written to the OCFL tree. Does anyone have a hunch what might be the problem here?

It sounds like one of the datastreams in druid:bb059dp5973 is missing a created date.

Yep, that's the bug. It's not sorting the datastream versions after it reads them from the FOXML and is relying on them appearing in the correct order. Thanks for digging into that. I can get that patched next week.

That would be great! Since our repository system allowed hand-editing of datastreams our data might exercise lots of corner cases.

@edsu This PR should fix this problem: https://github.com/fcrepo-exts/migration-utils/pull/182
Are you able to try it?

Yes, it worked! I see the object that failed above was able to be written.


DRT: Increase in DRCs after Detailed Routing when using cts NDR option

Describe the bug
When activating the clock NDR option on cts there is an increase on the DRCs after Deteiled Routing.
Expected Behavior
NO increase in the DRCs and the NDR rules respected.
Environment
kernel: Linux 5.15.0-113-generic
os: Ubuntu 20.04.6 LTS (Focal Fossa)
cmake version 3.24.2
-- The CXX compiler identification is GNU 9.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OpenROAD version: v2.0-14515-g65e6e329e
-- System name: Linux
-- Compiler: GNU 9.4.0
-- Build type: RELEASE
-- Install prefix: /usr/local
-- C++ Standard: 17
-- C++ Standard Required: ON
-- C++ Extensions: OFF
-- The C compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found Python: /usr/bin/python3.8 (found version "3.8.10") found components: Interpreter
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test C_COMPILER_SUPPORTS__-Wall
-- Performing Test C_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall
-- Performing Test CXX_COMPILER_SUPPORTS__-Wall - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test C_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-array-bounds - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test C_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-nonnull - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test C_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-maybe-uninitialized - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format-overflow - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-variable - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-function - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test C_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-write-strings - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test C_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-sign-compare - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test C_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-deprecated - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test C_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-c++11-narrowing - Failed
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register
-- Performing Test C_COMPILER_SUPPORTS__-Wno-register - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-CMake Warning (dev) at src/sta/CMakeLists.txt:33 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'USE_TCL_READLINE'.
This warning is for project developers.  Use -Wno-dev to suppress it.

register
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-register - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format
-- Performing Test C_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-format - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test C_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-reserved-user-defined-literal - Failed
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive
-- Performing Test C_COMPILER_SUPPORTS__-fpermissive - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive
-- Performing Test CXX_COMPILER_SUPPORTS__-fpermissive - Success
-- Performing Test C_COMPILER_SUPPORTS__-x
-- Performing Test C_COMPILER_SUPPORTS__-x - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-x
-- Performing Test CXX_COMPILER_SUPPORTS__-x - Failed
-- Performing Test C_COMPILER_SUPPORTS__c++
-- Performing Test C_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__c++
-- Performing Test CXX_COMPILER_SUPPORTS__c++ - Failed
-- Performing Test C_COMPILER_SUPPORTS__-std=c++17
-- Performing Test C_COMPILER_SUPPORTS__-std=c++17 - Failed
-- Performing Test CXX_COMPILER_SUPPORTS__-std=c++17
-- Performing Test CXX_COMPILER_SUPPORTS__-std=c++17 - Success
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test C_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable
-- Performing Test CXX_COMPILER_SUPPORTS__-Wno-unused-but-set-variable - Success
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- TCL readline library: /usr/lib/x86_64-linux-gnu/libtclreadline.so
-- TCL readline header: /usr/include/x86_64-linux-gnu
-- Found SWIG: /usr/bin/swig (found suitable version "4.1.0", minimum required is "4.0")
-- Using SWIG >= 4.1.0 -flatstaticmethod flag for python
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0")
-- boost: 1.80.0
-- Found Python3: /usr/include/python3.8 (found version "3.8.10") found components: Development Development.Module Development.Embed
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- spdlog: 1.8.1
-- Found BISON: /usr/bin/bison (found version "3.5.1")
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- STA version: 2.5.0
-- STA git sha: 1c7f022cd0a02ce71d047aa3dbb64e924b6efbd5
-- System name: Linux
-- Compiler: GNU 9.4.0
-- Build type: RELEASE
-- Build CXX_FLAGS: -O3 -DNDEBUG
-- Install prefix: /usr/local
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- SSTA: 0
-- Found SWIG: /usr/bin/swig (found suitable version "4.1.0", minimum required is "3.0")
-- STA executable: /home/arthur/work/SOpenROAD/tools/OpenROAD/src/sta/app/sta
-- Found re2: /opt/or-tools/lib/cmake/re2/re2Config.cmake (found version "9.0.0")
-- Found Clp: /opt/or-tools/lib/cmake/Clp/ClpConfig.cmake (found version "1.17.7")
-- Found Cbc: /opt/or-tools/lib/cmake/Cbc/CbcConfig.cmake (found version "2.10.7")
-- Found SCIP: /opt/or-tools/lib/cmake/scip/scip-config.cmake (found version "8.0.1")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- GUI is enabled
-- Charts widget is not enabled
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: serialization
-- Could NOT find VTune (missing: VTuneNumber of processor cores: 4
_LIBRARIES VTune_INCLUDE_DIRS)
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found suitable version "1.80.0", minimum required is "1.78")
-- TCL library: /usr/lib/x86_64-linux-gnu/libtcl.so
-- TCL header: /usr/include/tcl/tcl.h
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: serialization system thread
-- Found Boost: /usr/local/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0")
-- Found Eigen3: /usr/local/share/eigen3/cmake/Eigen3Config.cmake (found version "3.4.0")
-- TCL readline enabled
-- Tcl Extended disabled
-- Python3 enabled
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmp.RIC2g316Nr

To Reproduce
Information was sent separately.
Relevant log output
No response
Screenshots
No response
Additional Context
No response

Fixed by https://github.com/The-OpenROAD-Project-private/OpenROAD/pull/2165


Tor fail after system update (again!!)

Hello, I have just performed a raspbian system update and TOR just failed and won't start anymore. I am on the latest raspiblitz release.
output from $journalctl -u tor@default
Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.002 [notice] Tor <IP_ADDRESS> running on Linux with Libevent 2.1.8-stable, OpenSSL 1.1.1c, Zlib 1.2.11, Liblzma 5.2.4, and Libzstd 1.3.8. Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.002 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.002 [notice] Read configuration file "/usr/share/tor/tor-service-defaults-torrc". Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.002 [notice] Read configuration file "/etc/tor/torrc". Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.016 [warn] Skipping obsolete configuration option 'PortForwarding' Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.020 [notice] Opening Socks listener on <IP_ADDRESS>:9050 Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.020 [notice] Opened Socks listener on <IP_ADDRESS>:9050 Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.020 [notice] Opening Control listener on <IP_ADDRESS>:9051 Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.020 [notice] Opened Control listener on <IP_ADDRESS>:9051 **Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.000 [warn] Couldn't open file for 'Log notice file /mnt/hdd/tor/notice.log': Read-only file system Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.000 [warn] Couldn't open file for 'Log info file /mnt/hdd/tor/info.log': Read-only file system Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.000 [notice] Closing partially-constructed Socks listener on <IP_ADDRESS>:9050 Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.000 [notice] Closing partially-constructed Control listener on <IP_ADDRESS>:9051 Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.000 [warn] Failed to parse/validate config: Failed to init Log options. See logs for details. Sep 28 23:31:43 redpenguin tor[674]: Sep 28 23:31:43.000 [err] Reading config failed--see warnings above. Sep 28 23:31:43 redpenguin systemd[1]<EMAIL_ADDRESS>Main process exited, code=exited, status=1/FAILURE Sep 28 23:31:43 redpenguin systemd[1]<EMAIL_ADDRESS>Failed with result 'exit-code'. Sep 28 23:31:43 redpenguin systemd[1]: Failed to start Anonymizing overlay network for TCP. Sep 28 23:31:43 redpenguin systemd[1]<EMAIL_ADDRESS>Service RestartSec=100ms expired, scheduling restart.** Sep 28 23:31:43 redpenguin systemd[1]<EMAIL_ADDRESS>Scheduled restart job, restart counter is at 1. Sep 28 23:31:43 redpenguin systemd[1]: Stopped Anonymizing overlay network for TCP. Sep 28 23:31:43 redpenguin systemd[1]: Starting Anonymizing overlay network for TCP...
Could be it a data disk failure? This is the 2nd time happening on a fresh raspiblitz install after weeks of running smoothly.
Thank you in advance.

What I can make out of this that 'Log info file /mnt/hdd/tor/info.log': Read-only file system would mean a disk problem indeed.
There are 2 common causes:

Power supply issue - should be sorted with separate power supply to the HDD like a powered USB Hub or Y-cable with a separate power supply. Upgrade to a better >3A PSU if not using one already.
Physical disk error - you will need a new HDD and start from scratch.

In any case if you can login via ssh run /home/admin/99checkUpdate.sh and select Update anyway to have the script guide you through an LND backup process.

Thank you for the answer.
I am not sure about the physical disk error, it's a quite new nano SSD (6 months old, Samsung). Can't I just delete info.log and let the tor service recreate it?
ps I can run normally the node without TOR.
Brad on the TG chat last time said the issue could be related to this, but I did not dig into:
https://imgur.com/GTtVKbv.png

Rpi4 or 3? Is it the Samsung T5 disk?
You could deactivate Tor from the menu and delete the whole Tor directory from the SSD.
When you reactivate it the directory will be recreated.

Rpi3, nope it is a Samsung SSD 860 msata/nano disk.
I have tried to delete the whole tor directory, unfortunately the problem persists.
These are the tor related packages that have been upgraded on the last system update.
tor-geoipdb:armhf (<IP_ADDRESS>-1~d10.buster+1, <IP_ADDRESS>-1~d10.buster+1)
tor:armhf (<IP_ADDRESS>-1~d10.buster+1, <IP_ADDRESS>-1~d10.buster+1)
I can try to revert them to previous versions, though.

Sep 30 12:15:06 redpenguin tor[682]: Sep 30 12:15:06.496 [notice] Opening Control listener on <IP_ADDRESS>:9051
Sep 30 12:15:06 redpenguin tor[682]: Sep 30 12:15:06.496 [notice] Opened Control listener on <IP_ADDRESS>:9051
Sep 30 12:15:06 redpenguin tor[682]: Sep 30 12:15:06.498 [warn] Error creating directory /mnt/hdd/tor/sys: No such file or directory
Sep 30 12:15:06 redpenguin tor[682]: Sep 30 12:15:06.498 [warn] Failed to parse/validate config: Couldn't create private data directory "/mnt/hdd/tor/sys"
Sep 30 12:15:06 redpenguin tor[682]: Sep 30 12:15:06.498 [err] Reading config failed--see warnings above.
Sep 30 12:15:06 redpenguin systemd[1]<EMAIL_ADDRESS>Main process exited, code=exited, status=1/FAILURE
Sep 30 12:15:06 redpenguin systemd[1]<EMAIL_ADDRESS>Failed with result 'exit-code'.
Disabling/renabling tor service does not create the tor folder on /mnt/hdd/ anymore. Any hint on this? I am gonna try unistalling tor packages and let the script handle it.

I believe I break my system-tor lol.
$sudo journalctl -u tor@default
-- Logs begin at Mon 2019-09-30 12:23:29 BST, end at Mon 2019-09-30 12:30:37 BST
-- No entries --
Do you suggest to re-flash a new raspiblitz install?

Sure that is the fallback when the settings are messed up.
Backup LND (if there are funds) with the method above and start with a clean 1.3 SDcard image which we know  is working.


APP_ENV breaks builds that already use that environment variable

[x] Tried the troubleshooting Wiki
[x] Followed the migration Wiki

Describe the bug
The title basically says it. The addition of the APP_ENV override is a breaking change for those of us who already had that variable defined for other purposes.
Expected behavior
I should have a way of overriding this variable if I don't want to use it or want to use another variable.
I've already created a fix for this, so I'm opening this issue because the template wanted me to do so before opening a PR.

Hello there! It's been several days since I opened this, and I think it's a pretty straightforward issue and PR. Anything I can do to help this move forward? Thank you.

sorry! I was AFK for a few months. I will be reviewing it and probably merging it within the next week

Baaaahh

okay I'm checking it now!!

To be honest, I think APP_ENV should still be used even if you're already using it. It's better to be consistent with a standard. However, I won't be surprised if you're not the only one who wants this. I want to continue to label it as an experimental feature available as a backup in the README

if you're okay with that, I'll merge the changes in, add the "fix", then tag you and if you're okay with that, I'll release the next version?

I'll do release tomorrow

In our case, we can't use APP_ENV because the enum that we use has different values from those accepted by dotenv.  At some point I might refactor this to use something else, but it's pretty consistent across our projects. I'm fine with however you want to release. Thanks.

okay! with that, we can safely close this :)


tf.math.abs has different results when running under CPU or GPU

Click to expand!
Issue Type
Bug
Have you reproduced the bug with TF nightly?
Yes
Source
source
Tensorflow Version
tf 2.11.0
Custom Code
Yes
OS Platform and Distribution
Linux Ubuntu 18.04
Mobile device
/
Python version
3.7.5
Bazel version
/
GCC/Compiler version
/
CUDA/cuDNN version
11.2
GPU model and memory
GeForce RTX 2080Ti
Current Behaviour?
tf.math.abs has different results when running under CPU or GPU.
The reproduction colab link is here: colab.
Standalone code to reproduce the issue
import tensorflow as tf

x = tf.constant([3.4e+307 + 0.j], dtype=tf.complex128)

with tf.device("/cpu:0"):
    a = tf.math.abs(x)
    print(a)

with tf.device("/gpu:0"):
    a = tf.math.abs(x)
    print(a)

Relevant log output
tf.Tensor([3.4e+307], shape=(1,), dtype=float64)
tf.Tensor([inf], shape=(1,), dtype=float64)

Hi @PengHaoYu ,
I have replicated the issue with TF2.12 and also tf-nightly versions.Please refer gist.
Needs to dig more into the issue and let you know. Thanks!

HI@PengHaoYU,
I executed the code using TensorFlow version 2.17, and it worked successfully.It works same under CPU and GPU,
Kindly refer to this gist for the details.
Thanks


Shell tests, continued.

Improvements as suggested in https://github.com/opentable/sous/pull/163.

Current coverage is 55.07% (diff: 100%)

Merging #164 into master will not change coverage

@@             master       #164   diff @@
==========================================
  Files           107        107
  Lines          5273       5273
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           2904       2904
  Misses         2123       2123
  Partials        246        246

Powered by Codecov. Last update 14831bf...3b75f0f


feat: Gleason's theorem

Unsure where to put the lemmas mem_coe_of_mem, ..., image_coe_eq_restrict_image - any ideas appreciated.

bors merge


[Bug] Only return ChatGPT

Hi, now only return the name "ChatGPT"
from gpt_lib import initGPT, initHuggingChat

chathead = initGPT(username ="email_pgt", password= "pass_gpt")

answer = chathead.interact("Hello, how are you today")

print(answer)

Hello @isMMa ,
After my issue with "2023/11/27 17:55:42 ERROR It is not possible to pass verification #46", I had the same problem as you: chathead.interact only returning the word "ChatGPT".
My opinion was that ChatGPT has become slower recently in displaying its response.
This is likely related to allocated resources versus the number of users, perhaps also a geographical issue.
In any case, this could explain why we only get the word "ChatGPT" in return.
I then added a pause of several seconds in the file 'chatgpt.py', function 'interact', after the line self.wait_until_disappear(By.CLASS_NAME, self.wait_cq):
        self.wait_until_disappear(By.CLASS_NAME, self.wait_cq)
        time.sleep(60)  # Modification Free-Max367
        answer = self.browser.find_elements(By.CLASS_NAME, self.chatbox_cq)[-1]
        logging.info('Answer is ready')
It works well now. I get a complete response.
Yes, 60 seconds is a lot, but I have no time constraints.
However, it's clearly not the right approach, and I am confident that @ugorsahin will find a much better automation method than what I have used here.
And, I will gladly beta-test the new release if @ugorsahin does not see this response issue on their end.
Thanks in advance.
Technical info: I'm using Windows 10 Entreprise version 22H2 + chrome-for-testing version 116.0.5845.96

Yes, now its works:

Hello @isMMa ,
On my end, it still doesn't work even with version 0.2.3.
To conduct the test, I ask ChatGPT to summarize a text of around 4000 characters.
In return, I get an empty text.
I still believe it's a response time issue with ChatGPT.

Hey,
The version 0.3.0 works perfectly on my end.
Thanks.


Better representation of Inline values

Signed-off-by: Hasan Turken<EMAIL_ADDRESS>

Codecov Report

Merging #28 into master will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master      #28   +/-   ##
=======================================
  Coverage   57.34%   57.34%
=======================================
  Files          10       10
  Lines         497      497
=======================================
  Hits          285      285
  Misses        187      187
  Partials       25       25

Flag
Coverage Δ

#unittests
57.34% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/controller/values.go
85.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 24d4676...3a36c98. Read the comment docs.


Replace double quotes with single in default user_agents.

In line with CodeIgniter style guide.

Already pending in pull #895.

Thanks Nick but looks like somebody else beat you to it.
That said, im not sure this is something we really need to be concerned with. The difference between the two is so incredibly trivial that it makes essentially no difference whatsoever unless you are concatenating/parsing strings, and even then the difference is moot.


Node#_important was deprecated. Use Node#raws.important

Version 1.0.0, running cssfmt test.css in command line return Node#_important was deprecated. Use Node#raws.important
It looks like it is only happening for .css file though, not .scss.

@dmnsgn Please gime me an example CSS code that error occurs.

@morishitter I used the input.css from the README

CSSfmt passed the test case in README example code, and I tried to it in my PC using CSSfmt v1.0.0.
But, no error occurs.

You mean this https://github.com/morishitter/cssfmt/blob/master/test/fixtures/readme.css ? Because it is using the SCSS syntax with mixin, @extend .
I started from scratch, installed cssfmt globally then I used the following CSS (taken from the README), ran cssfmt input.css and then the error occurs (tested on 3 different platforms).
      @media screen and (    min-width :699px)
 {.foo    +       .bar,.hoge{
    font-size   :12px      !   important   ;  ~       .fuga     {
      padding      : 10px       5px;
   color:green;
 >p

 {
        line-height             : 1.5          ;
      }}}
     }                /* comment for .class, #id */

.class,           #id
 {     color       : blue;

  border        :solid  #ddd                1px}

I understarnd. Thank you for your report. I will fix this :)


Dropdown field with surrounding text

The following would be a use case: Check for the following only if text matches ['all','any'] of the following: <input>, where ['all','any'] is an array for a dropdown and <input> is a text field. I'm likely going to have to put this into a single group.
Any idea how to do this?

In other words, how can I use ng2-dynamic-forms to create complex filter forms?

@kahboom Hi!
I'm not sure if I fully understand your use case.
I guess you already had a look at the chapter on (Related Form Controls)[https://github.com/udos86/ng2-dynamic-forms#related-form-controls].
What exactly do you mean by complex filter form?

Thanks for the prompt response @udos86 ! I was looking for a way to implement something like this using Bootstrap styling (including inline forms, groups, etc) and the ability to add field groups dynamically:

It looks like ng2-dynamic-forms does handle this, although I'd need to see how to use Bootstrap styling proprely. It turns out it's a project-specific thing (the project I'm working on) that limits my ability to do that, so I'll need to refactor that in the future. Since I'm short on time I think for this specific form I'll just use regular forms and keep using ng2-dynamic-forms for all other forms in the project though. Again, many thanks! 👍

@kahboom Great to hear! Thanks for using the lib!

Okay, reopening for a quick, related question, since I decided to go back to using ng2-dynamic-forms after all for this filter lol.. I have something like this at the moment, using ui-bootstrap:

I would like to wrap the "of the following" text in the top form group to be inline with the select model (if you see the screenshot of the design I posted above, it's number 2 of the fields), so that it's not within the field. Is this possible to do with the model file, or what would you suggest?
It uses the following model:
export const BASIC_FILTER_MODEL = [
  new DynamicFormGroupModel({
    id: 'filterSettingsGroup',
    group: [
      new DynamicSelectModel<string>(
        {
          id: 'matchSelect',
          label: 'Continue only if incoming data match ',
          options: Observable.of([
            {
              label: 'ALL of the following',
              value: 'all',
            },
            {
              label: 'ANY of the following',
              value: 'any',
            },
          ]),
          value: 'all',
        },
        {
          element: {
            //container: 'form-group',
            label: 'control-label pull-left',
          },
          grid: {
            control: 'col-xs-3',
          },
        },
      ),
    ],
  }),

  new DynamicFormGroupModel({
    id: 'rulesGroup',
    group: [
      new DynamicFormArrayModel(
        {
          id: 'rulesFormArray',
          initialCount: 5,
          groupFactory: () => {
            return [
              new DynamicInputModel(
                {
                  //hint: 'Just a sample help text',
                  id: 'dataToBeFiltered',
                  maxLength: 51,
                  placeholder: 'Status.Text',
                  suffix: 'Browse...', // This is just a suffix; this whole field needs to change
                  /*
                  validators: {
                    required: null,
                    maxLength: 5,
                  },
                  errorMessages: {
                    //required: '{{ label }} is required',
                    required: 'Please select data to filter',
                    maxLength: 'Max character count is 5',
                  },
                  */
                },
                {
                  element: {
                    container: 'form-group col-xs-3',
                    //label: 'control-label',
                  },
                  grid: {
                    control: 'input-group',
                    //errors: 'col-xs-offset-3 col-xs-3',
                  },
                },
              ),
              new DynamicSelectModel<string>(
                {
                  id: 'conditions',
                  options: Observable.of([
                    {
                      label: 'Contains',
                      value: 'contains',
                    },
                    {
                      label: 'Does Not Contain',
                      value: 'does-not-contain',
                    },
                    {
                      label: 'Matches Regex',
                      value: 'matches-regex',
                    },
                    {
                      label: 'Does Not Match Regex',
                      value: 'does-not-match-regex',
                    },
                    {
                      label: 'Starts With',
                      value: 'starts-with',
                    },
                    {
                      label: 'Ends With',
                      value: 'ends-with',
                    },
                  ]),
                  value: 'contains',
                },
                {
                  element: {
                    container: 'form-group col-xs-3',
                    label: 'control-label',
                  },
                  grid: {
                    control: 'input-group',
                  },
                },
              ),
              new DynamicInputModel(
                {
                  id: 'valueToCheckFor',
                  placeholder: 'e.g. Red Hat',
                },
                {
                  grid: {
                    container: 'input-group',
                  },
                },
              ),
            ];
          },
        },
        {
          element: {
            container: 'form-inline form-array',
          },
          grid: {
            //control: 'col-xs-12',
          },
        },
      ),
    ],
  }),
];

With the following HTML:
<form [formGroup]="formGroup">
        <dynamic-form-bootstrap-control *ngFor="let controlModel of basicFilterModel"
                                        [group]="formGroup"
                                        [model]="controlModel"
                                        (blur)="onBlur($event)"
                                        (change)="onChange($event)"
                                        (focus)="onFocus($event)">
          <ng-template modelId="rulesFormArray"
                       let-context="context"
                       let-index="index">

              <button type="button"
                      class="btn btn-danger btn-sm rounded"
                      (click)="remove(context, index)">✕
              </button>
              <button type="button"
                      class="btn btn-default btn-sm rounded"
                      (click)="move(context, index, -1)">▲
              </button>
              <button type="button"
                      class="btn btn-default btn-sm rounded"
                      (click)="move(context, index, 1)">▼
              </button>
              <button type="button"
                      class="btn btn-success btn-sm rounded"
                      (click)="insert(context, index + 1)">+
              </button>

          </ng-template>
        </dynamic-form-bootstrap-control>
      </form>

Also, any general suggestions you can provide would be very helpful. Doing a demo using your wonderful library this Friday. :) Thanks again.

And, yes, I need to update that prefix input field to instead be a file uploader :joy:. Really leaving things for last minute here.

I had a few other questions, which of course I can't think of at the moment. One was that for some reason, those buttons that appear (the up/down, add, remove buttons for each row) only appear when the mouse clicks anywhere on the page, or focuses on the form. I will do some debugging on this, but was wondering if you had any ideas off the top of your head as to why this could be happening. Thanks in advance!

@kahboom Thanks for your kind words!
In order to place your text inline with but outside the actual select control I'd suggest that you simply use a custom template:
<ng-template modelId="matchSelect">

of the following

</ng-template>

Unfortunately I cannot verify your rendering issues in my samples. @rernens reported some similar problems (see #431) but at the moment I'm not able to reconstruct that behavior.

Thank you @udos86! Especially for the quick response. Regarding that last point, I think it's an inheritance issue with events not propagating properly, likely unrelated to the lib, but will report back if I find anything. I'll give your suggestion for custom templates a shot today.

@kahboom That's great! I appreciate any help here. Good Luck!


Removes color from tail output for --no-color

Pull request

What this PR does / why we need it
PR removed the ANSI color codes from tail output when --no-color is set as true
Which issue(s) this PR fixes

Fixes #368
Describe testing done for PR

Added unit test cases, also validated in local cluster for tail commands and create with tail.
Additional information or special notes for your reviewer


Protocol error (Bad Record MAC) when using AES256-SHA cipher

I'm having an issue when trying to connect a TLSe client to an openssl server (version 1.0.1e). The issue happens when the selected cipher is TLS_RSA_WITH_AES_256_CBC_SHA (0x0035).
The problem is easily reproducible by spawning an openssl server with:
openssl s_server -tls1 -debug -cert cert.pem -cipher AES256-SHA

And then connecting to this server with tlsclienthello sample.
SSL library drops the following message:
140183811917640:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:s3_pkt.c:486:

If using openssl s_client for connecting to the server, everything works normally.
I'm attaching tlse-bad-rec-mac.zip, a (zipped) pcap with the protocol exchange (client is <IP_ADDRESS>).

Seems that non-ephemeral ciphers are broken when using TLS 1.0 and TLS 1.1 (fine on TLS 1.2). I'll take a look.

It should now be fine. Can you confirm ? It was a problem with md5 and sha1 mac sizes used in PRF initialization.

Just checked with openssl 1.0.1l ... still doesn't work.

Ok, I think is OK now. Can you check the latest version ?

Yep, just re-tested and it seems to be working fine now. Thanks for the help!
(PS: I found another issue during latest tests, I'm opening a PR with a small fix for it)


Treat Records pessimistically

🚀 Feature request
Treat Records pessimistically
Current Behavior
Currently, record.fromFoldable('..') returns a Record that can fail at runtime due to the fact one may access the Record with a key that don't have an entry in the Record.
So even if using fp-ts with all its type checking, I am still able to get an undefined is not a function.
That's not acceptable IMHO.
Desired Behavior
record.fromFoldable('..') and all related functions that generate Records must return a Partial<Record<..,..>> if it is not provably safe to do otherwise (we may find a nicer Alias for that purpose).
Also all functions that may transform from a Record<..,..> or Partial<Record<..,..>> to another data-structure must try to ease the developper experience by not exposing Partials or undefined types if it is typesafe to do so (like lookup or toArray which should not reflect the input Record was Partial or not).
Suggested Solution
I propose, if @gcanti agree on the goal, to make all the changes related to this feature as a proof of concept.
Who does this impact? Who is this for?
This is for all users.
Describe alternatives you've considered
Killing myself (metaphorically).
Additional context
Your environment
fp-ts, all versions.

Hi @sledorze , could you please provide your desired signature for fromFoldable? I'm not sure I get what is the exact proposal here.
Re "unsafe index access" in general, since it isn't something that will get any stricter support from TS, probably the easiest path is to find/create an eslint plugin disallowing it for any object with a non-finite index signature, including Array? (or just disallowing it altogether probably)

@gcanti @giogonzo I've been into the rabbit hole of trying to make a POC.
Thats actually another abstraction IMHO.
https://github.com/gcanti/fp-ts/tree/977
Maybe good for fp-ts-contrib to stress the idea?

Note that I see some inference issues with literal Records ({ [string] : A } vs Record<string, A>).
Using overloadings seems not to solve it completely..
Not without conflating Partials with Record that actually has some entries with real undefined values..

Desired:

Is it correct to say that what you really want is to avoid infinite domain records OR treat them as having optional keys? Something like
type MyRecord<K extends string, A> = string extends K
  ? Record<K, A | undefined>
  : Record<K, A>;

type A = MyRecord<string, number>; // { [x: string]: number | undefined; }
type B = MyRecord<"foo", number>; // { foo: number; }

Record<K, A | undefined> and Partial<Record<K, A>>

when is this distinction useful?

No, what I want is to avoid:
const xs: [string, string][] = [] record.fromFoldable(xs)['a'].length // Crash
This is the simplest use case.
The impossibility (AFAIK) to differentiate between Partial<Record<K, A>> and Record<K, A | undefined> is the crux of having a sound implementation.

probably the easiest path is to find/create an eslint plugin disallowing it for any object with a non-finite index signature, including Array? (or just disallowing it altogether probably)

I've started poking at exactly this approach in https://github.com/danielnixon/total-functions

I think this Issue is not actionnable ATM with current Typescript limitations.
Closing it for now


Metadata: Discovering derived type with self ref relationship before base causes exception

I am migrating from .net core 1.1 to 2.0 and I meet a error when I add entityframeworkcore migrations.
I have two models.
public class TestNode
{
    public int Id { get; set; }

    public TestNode Parent { get; set; }

    public List<TestNode> Children { get; set; } = new List<TestNode>();
}

public class TestCase : TestNode
{
     public string Description { get; set; }
}

And I configured an ApplicationDbContext.
public class ApplicationDbContext : DbContext
{
    public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
            : base(options)
    {
    }

    public DbSet<TestNode> TestNodes { get; set; }

    public DbSet<TestCase> TestCases { get; set; }
}

Then I executed add-migration init in console. Following error occurred
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0]
      User profile is available. Using 'C:\Users\karl.gong\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
System.InvalidOperationException: Nullable object must have a value.
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Nullable`1.get_Value()
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.DetachRelationship(ForeignKey foreignKey)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder.HasBaseType(EntityType baseEntityType, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.DerivedTypeDiscoveryConvention.Apply(InternalEntityTypeBuilder entityTypeBuilder)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeAdded(InternalEntityTypeBuilder entityTypeBuilder)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.RunVisitor.VisitOnEntityTypeAdded(OnEntityTypeAddedNode node)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.OnEntityTypeAddedNode.Accept(ConventionVisitor visitor)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionVisitor.Visit(ConventionNode node)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionVisitor.VisitConventionScope(ConventionScope node)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionBatch.Run()
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionBatch.Dispose()
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.RelationshipDiscoveryConvention.DiscoverRelationships(InternalEntityTypeBuilder entityTypeBuilder)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.RelationshipDiscoveryConvention.Apply(InternalEntityTypeBuilder entityTypeBuilder)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeAdded(InternalEntityTypeBuilder entityTypeBuilder)
   at Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.OnEntityTypeAdded(InternalEntityTypeBuilder entityTypeBuilder)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.Model.AddEntityType(EntityType entityType)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.Model.AddEntityType(Type type, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder.Entity(TypeIdentity type, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder.Entity(Type type, ConfigurationSource configurationSource)
   at Microsoft.EntityFrameworkCore.ModelBuilder.Entity(Type type)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
   at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.FindSets(ModelBuilder modelBuilder, DbContext context)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
   at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.Customize(ModelBuilder modelBuilder, DbContext context)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.<>c__DisplayClass5_0.<GetModel>b__0(Object k)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.GetModel(DbContext context, IConventionSetBuilder conventionSetBuilder, IModelValidator validator)
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
   at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkServicesBuilder.<>c.<TryAddCoreServices>b__7_1(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(ScopedCallSite scopedCallSite, ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(IServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.<>c__DisplayClass22_0.<RealizeService>b__0(ServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.EntityFrameworkCore.DbContext.get_DbContextDependencies()
   at Microsoft.EntityFrameworkCore.DbContext.get_InternalServiceProvider()
   at Microsoft.EntityFrameworkCore.DbContext.Microsoft.EntityFrameworkCore.Infrastructure.IInfrastructure<System.IServiceProvider>.get_Instance()
   at Microsoft.EntityFrameworkCore.Infrastructure.AccessorExtensions.GetService[TService](IInfrastructure`1 accessor)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(Func`1 factory)
   at Microsoft.EntityFrameworkCore.Design.Internal.DbContextOperations.CreateContext(String contextType)
   at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.AddMigration(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigrationImpl(String name, String outputDir, String contextType)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigration.<>c__DisplayClass0_1.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
Nullable object must have a value.

The trick is that if I change the line public DbSet<TestCase> TestCases { get; set; } in ApplicationDbContext to public DbSet<TestCase> XXXXs { get; set; }, IT WORKS!
Is TestCases a reserved word?
Thanks,
Karl

The error still occurred even I change the name.

The base class TestNode has one-to-many self-ref relationship.
If TestCase derived type is discovered before TestNode then it throws.

Impact: Self-referencing relationships cause exceptions if a derived type is configured
Risk: Low, the fix only affects the negative scenario

Deriving from a self-referencing entity is an important pattern for us. Is there a recommended work around?

@SpiraMira The workaround is to force the derived type to be discovered last. This could be accomplished in several ways:

Renaming TestCases to something like ZTestCases as sets are discovered in alphabetical order.
Removing the derived set property and configuring the entity type in OnModelCreating.
Adding a NotMapped annotation on the self-ref navigation and configuring it in OnModelCreating.

@AndriySvyryd Thanks for the quick turn-around.  This will help.
Regards.

This patch bug is approved for the 2.0.x patch. Please send a PR to the feature/2.0.1 branch and get it reviewed and merged. When we have the rel/2.0.1 branches ready please port the commit to that branch.

still getting this issue even though I tried the workarounds mentioned in the thread. Any plans for the patch release ?

Hi, we have a public test feed that you can use to try out the ASP.NET/EF Core 2.0.3 patch!
To try out the pre-release patch, please refer to the following guide:

https://github.com/aspnet/Announcements/issues/274 (info)
https://github.com/aspnet/Home/issues/2254 (discussions and questions)

We are looking for feedback on this patch. We'd like to know if you have any issues with this patch by updating your apps and libraries to the latest packages and seeing if it fixes the issues you've had, or if it introduces any new issues. If you have any issues or questions, please reply on this issue to let us know as soon as possible.
Thanks,
Eilon


docs: change references slack

Pull Request Template

Closes #5090

READMEs badges and references
Contributing docs
Other doc references
GitHub discussions

Type of change

Documentation update

How Has This Been Tested

sphinx-autobuild

mkdocs-serve

Checklist

I made corresponding changes to the documentation

@sdiazlor , the Join Discord buttons look a bit weird to me. Would it be possible to just dhow the discord logo? or only a square logo instead of the entire text?

@davidberenstein1957 WDYT? Writing Discord directly and using only the icon logo I believe is more consistent with the rest of buttons

Looks great!


Spring Cloud AWS Messaging > 3.0

Type: Spring Cloud AWS Messaging > 3.0
Is your feature request related to a problem? Please describe.
According to the Spring blog post published on May 2, 2023 (link: https://spring.io/blog/2023/05/02/announcing-spring-cloud-aws-3-0-0), the new version of Spring Cloud AWS, version 3.0.0, has been announced.
However, it appears that the messaging dependency for version 3.0.0 is currently unavailable in the Maven repository (link: https://mvnrepository.com/artifact/io.awspring.cloud/spring-cloud-aws-messaging).
I am interested in knowing the ETA for the messaging dependency to be available. At the moment we are having problems trying to upgrade services that require the messaging dependencies.
Describe the solution you'd like
Be able to use AWS Messaging > 3.0 because this is the only way to update services using JDK17, Spring 6 and Spring Boot 3.
io.awspring.cloud:spring-cloud-aws-messaging:3.0.0
Describe alternatives you've considered
This is a blocker. There is no alternative
Additional context

Hi @rl1m4, as part of the changes made for Spring Cloud AWS 3.0, the spring-cloud-aws-messaging module has been replaced with two separate modules supporting SQS and SNS integrations. There are corresponding libraries and starter dependencies for each. The starter dependencies are documented in the 3.0 reference guide - https://docs.awspring.io/spring-cloud-aws/docs/3.0.0/reference/html/index.html#getting-started.
Neil.

Thank you for the prompted answer. Cheers

@rl1m4 unfortunately we did not manage to provide a list of breaking changes and a migration guide. 3.0 in terms of SQS is like a new framework, so I would recommend really going through the new reference docs in details.


[docs] Update misspelled "Interactive" in Tooltip Demo

[ ] I have followed (at least) the PR section of the contributing guide.

FIX: #13424

@imjaroiswebdev It's a great first pull request on Material-UI 👌🏻. Thank you for working on it!


lifetime errors using db with axum

struct AppState {
    db: Arc<Db>,
    tera: Tera,
}

error: implementation of `Send` is not general enough
  --> examples/toasty/main.rs:72:21
   |
72 |         .route("/", get(root))
   |                     ^^^^^^^^^ implementation of `Send` is not general enough
   |
   = note: `Send` would have to be implemented for the type `&Schema`
   = note: ...but `Send` is actually implemented for the type `&'0 Schema`, for some specific lifetime `'0`

error: implementation of `IntoSelect` is not general enough
  --> examples/toasty/main.rs:72:21
   |
72 |         .route("/", get(root))
   |                     ^^^^^^^^^ implementation of `IntoSelect` is not general enough
   |
   = note: `IntoSelect<'_>` would have to be implemented for the type `Select<'_, Like>`
   = note: ...but `IntoSelect<'0>` is actually implemented for the type `Select<'0, Like>`, for some specific lifetime `'0`

#[axum::debug_handler]
async fn root(State(state): State<AppState>) -> Result<Html<String>, AppError> {
    let context = Context::new();
    let tera = state.tera;
    let db = state.db;
    let user = User::find_by_name("Guy").first(&db).await?.unwrap();
    let mut bookmarks = Vec::new();
    let mut q = user.likes().all(&db).await.unwrap();
    while let Some(like) = q.next().await {
        bookmarks.push(like.unwrap().bookmark().find(&db).await.unwrap());
    }
    println!("{name} likes {bookmarks:?}", name = user.name);
    let render = tera.render("index.tera.html", &context)?;
    Ok(Html(render))
}

Thanks. I know what is going on. It is going to take a bunch of reshuffling lifetimes internally, but I should have a fix soon.

This error can also be triggered with a simple tokio::spawn.
    let db = Db::new(schema, driver).await;
    tokio::spawn(async {
        User:: find_by_name("foo"). first(&db).await.unwrap();
    });

Glad you know what's going on. Can't wait to see what's the fix :)

I isolated it as a test and the fix will be merged soon: https://github.com/tokio-rs/toasty/pull/21/files#diff-e669e107b34d2d12ea757a08b7386701e19cc367aeb5fee09ff0912e7d2b8631

I haven't verified that your original repro works yet. If it doesn't, there is a second (or more) layer to this.

This error can also be triggered with a simple tokio::spawn.
    let db = Db::new(schema, driver).await;
    tokio::spawn(async {
        User::find_by_name("foo").first(&db).await.unwrap();
    });

Glad you know what's going on. Can't wait to see what's the fix :)

For this, you need to throw Db in an Arc.

I just meant the error was reproducible using the spawn instead of trying to reproduce with the whole axum framework, but it's all good now, working fine with the fix :)


Cursor decorator

Display a blinking cursor at the given offset. Implement hit-testing the text to retrieve a cursor offset.

Current middleware idea will need a pre/post draw method that gets the position and text/amount of space to draw.

Example is working, but not ready to become a stable plugin yet.


Customize Build Status Message

From what I can tell it looks like updating that build status is pretty specific to test output (based on what I can see here and here). That is what makes the build status appear like:

I haven't been able to find a way to set that status to something more custom - for example I have a separate build that runs static analysis and uses the Violations plugin to report violations in the Jenkins UI - it would be nice if I could have the build status updated to "Build finished. X violations found." or something along those lines.
I've tried playing around with having my job set a custom environment variable and trying to use that but I couldn't seem to get that to work. Is there some way to already accomplish this? Is there some way to integrate this plugin with the Violations plugin to make that information available?
I'm open to any suggestions of things that may already accomplish this. I would also be willing to submit a PR to make this customization possible (however I may need some direction on the best way of accomplishing it).
Thank you!

Hi - I'm helping out with this repo. I'm not sure of the state on Mar 17; however, this kind of customization should be available in the downstream jobs pattern that's described in the README today. I'll close this but feel free to re-open if that's not helping.


java.lang.IllegalArgumentException: ByteBuffer is not a valid flatbuffer model issue

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on a mobile device: Samsung S5
TensorFlow installed from (source or binary): Binary
TensorFlow version: 1.12
Python version: 3.5
Installed using virtualenv? pip? conda?: conda
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
I have created my own architecture, on tensorflow and I have used Toco to convert to .tflite, However the android app still throws the following error: "java.lang.IllegalArgumentException: ByteBuffer is not a valid flatbuffer model."
Provide the exact sequence of commands / steps that you executed before running into the problem
'''
AssetFileDescriptor fileDescriptor = activity.getAssets().openFd(MODEL_PATH);
FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
FileChannel fileChannel = inputStream.getChannel();
long startOffset = fileDescriptor.getStartOffset();
long declaredLength = fileDescriptor.getDeclaredLength();
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
'''
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Apologies for the delay in response. Is this still an issue for you?
Can you please take a look at duplicate #23628? Thanks!

Hi there, I'm trying to convert a Tensorflow 1.12 / Keras model with TFLiteConverter.from_keras_model_file. I've got no issue doing this on Colaboratory (tf 1.12), but when I run my model I encounter the dreaded error :
"Caused by: java.lang.IllegalArgumentException: ByteBuffer is not a valid flatbuffer model" error. with firebase-ml-model-interpreter:16.2.4
The model structure is the following :
Input(884)
BatchNormalization()
Reshape(52,17)
LSTM(256, unroll=True, return_sequences=True)
LSTM(256, unroll=True)
BatchNormalization()
Dense(260)
I previously successfully converted and used similar models, but without Reshape and (unrolled) LSTM layers.
Any advice on version or changes I should make ?
Thank you very much in advance.

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!
@NicolasVidal Can you please post a new issue explaining your problem if it still persists? thanks

I'm having the same issue, converted the model with from_keras_model_file, tried to load it in Android but gives me this error:
java.lang.IllegalArgumentException: Contents of /conv.tflite does not encode a valid TensorFlowLite model: Could not open '/conv.tflite'.The model is not a valid Flatbuffer file


init

Hello ! I just wanted to pop in to discuss a certain point.
In Actor-Critic algorithms, it is often the case that we must sample from the trajectory to update the critic, and then a second time to update the actor. It may also be so for model-based methods if there's a sampling to train the model.
Depending on the algorithm, it may be that the critic needs a different kind of batch than the actor/policy. I think of retrace that needs N (states, actions, rewards) batched to update the critic, but the actor needs a single state. So that means that the Sampler should be different. It may be nice to think of how to implement the possibility to specify multiple samplers. Maybe Sampler could allow for a Dict like so:
:policy => BatchSampler
:qnetwork => NStepBatchSampler (3 steps)
:model => NStepBatchSampler (100 steps)

And then algorithms would call e.g. inds, batch = traj.sampler[:policy](traj) when updating the policy but inds, batch = traj.sampler:qnetwork` when updating a qnetwork.
Each algorithm could check that the required sampler is present in the Dict during the check phase.

Thanks for providing early feedback.
Yes, it should be easy to support. We just need a meta sampler 😉


WebXR camera positioning API

It should be possible to position the XR camera in any 3D position in space.
The current state allows a developer to re-position the webxr container (the parent of the camera and the controllers), but in order to do that correctly the user must also know the height of the user and other factors. There is also an issue when querying the current camera's position and direction, as the camera itself will always return 0,0,0 as its local position due to the way the rig-cameras are working.
This is a proposed API extension to the WebXR camera that will be able to position the container based on floor meshes, ignoring the user's current height. The camera should be the "single point of truth" when it comes to its own position.
The API will allow:

Simple (re)positioning of a user to a point of reference, with an optional floor / height map to use for height:

function setPosition(pointInSpace: Vector3, pointOfReference?: Vector3 | Node, options? : {
  space: Space = Space.LOCAL,
  setHeightFromXRFrame: boolean = true,
  // further options TBD
}): void;

Get the camera's current global and local position using the XFrame information sent by XR (average of both eyes)
The ability to query the global position of a single eye
The ability to check the camera's current direction
Active position-copy from native (non-XR) to XR cameras and vice versa

@nbduke - please comment if anything is missing here

So if I call setPosition(point, Vector3.Zero(), { space: Space.LOCAL }), is that just the same as setPosition(point, undefined, { space: Space.WORLD })?

@nbduke - Sorry for a late answer!
Yes, world coordinates will be including the container, local will change the camera alone. Internally, both will change the container but either relative to the current position or not.

Teleporting (position changes of the user) will be done using https://www.w3.org/TR/webxr/#xrreferencespace-interface
the setPOsition function will create a new relative reference space.

merged and tested

@RaananW
Using xr.baseExperience.camera.leftCamera and xr.baseExperience.camera.rightCamera does not seem to work anymore. Which is strange, because xr.baseExperience.camera.rigCameras returns an Array with two cameras, but xr.baseExperience.camera.rigCameras.length returns 1 :confused: .
https://www.babylonjs-playground.com/#LUYXFU

Hi @stefanvanherwijnen , just got back and looking into this.
About the first example, this is what I get:

Which is what is expected right before the first frame was rendered.
I understand that you want a kind of callback that will be triggered when everything was set, and I agree that it is important.
This - https://www.babylonjs-playground.com/debug.html#LUYXFU#3 should be working, but due to the fact that I use the first frame to update the camera's position from the old camera, it doesn't . But I believe I have found a way to make it work correctly. Want to create a github issue so we can track it internally?
Thanks!

Thanks, I opened a new issue.
You're right that I'm looking for a callback that is triggered when everything is set, Before the recent changes the IN_XR state was such that everything XR related was ready and working.


Add some debug message for compaction

Add some debug message for compaction

Codecov Report
Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (73f2a37) 0.00% compared to head (15d4273) 18.34%.

Files
Patch %
Lines

...java/com/github/ambry/store/CompactionManager.java
0.00%
4 Missing :warning:

...github/ambry/store/StatsBasedCompactionPolicy.java
0.00%
2 Missing :warning:

...ava/com/github/ambry/store/BlobStoreCompactor.java
0.00%
1 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2681       +/-   ##
=============================================
+ Coverage          0   18.34%   +18.34%
- Complexity        0     2789     +2789
=============================================
  Files             0      816      +816
  Lines             0    69773    +69773
  Branches          0     8419     +8419
=============================================
+ Hits              0    12800    +12800
- Misses            0    55784    +55784
- Partials          0     1189     +1189

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Non-root building/signing on Ubuntu

Cannot non-root build on Ubuntu 14.04 and 16.04 due to the need use
proc = salt.utils.vt.Terminal
in order to use reprepro and debsign, and internally this is causing pool files to have owner root, for example:
oot@ip-10-27-37-59:/srv/ubuntu/2018.3.3/pkgs/py2/ubuntu1604/amd64# l pool/main/i/ioflo/
total 388K
-rw-r--r-- 1 ubuntu ubuntu 161K Dec  3 18:28 python-ioflo_1.5.0-1_all.deb
drwxr-xr-x 3 ubuntu ubuntu 4.0K Dec  3 18:28 ..
-rw-r--r-- 1 root   root   1.4K Dec  3 18:29 ioflo_1.5.0-1.dsc
-rw-r--r-- 1 root   root   207K Dec  3 18:29 ioflo_1.5.0.orig.tar.gz
-rw-r--r-- 1 root   root    960 Dec  3 18:29 ioflo_1.5.0-1.debian.tar.xz
drwxr-xr-x 2 ubuntu ubuntu 4.0K Dec  3 18:29 .
This is not an issue with Ubuntu 18.04 (or on Debian 8 & 9) where the following command cmd.retcode can also be used:
935                 if grains['os'] in ['Debian']:
936                     log.debug("DGM make_repo debian gpg2 reprepro cmd '{0}'".format(cmd))
937                     retrc |= salt['cmd.retcode'](cmd, runas=runas, cwd=repodir, use_vt=True, env=env)
938                 else:
939                     log.debug("DGM make_repo ubuntu gpg2 reprepro cmd '{0}'".format(cmd))
940                     try:
941                         proc = salt.utils.vt.Terminal(
Quickest fix is to just continue to use root to build on Ubuntu, noting that it was only Debian 9 (at this time) which needed non-root to use gpg-agent for signing, etc.

Blocked since don't have a solution at this time, and in April Ubutnu 14.04 EOL's at which point can consider a 16.04 only fix.


Fix radiusMinPixels and radiusMaxPixels for METER_OFFSETS in scatterplot

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: huaruiwu:x: drivkinYou have signed the CLA already but the status is still pending? Let us recheck it.

@huaruiwu Thanks for the PR!
Yes, this is a bug and your change solves the problem. However, it might be better to have this check on projection mode be implemented in shaderlib/project.glsl.js. This will solve this problem also for other layers and make the code clean.
Could you update your change?

Glad to be of help!
great suggestion. I will take a look and try to make the appropriate changes.

@shaojingli You were right, looks like there was some weird code causing bugs in project.glsl.js. I think I've fixed it. Could you check it over? Thanks!

This doesn't look right to me... Ib can you review it? @ibgreen

We need to be very careful with changing project functions.

The idea was to convert things to meters, so that the shader can do math in meters regardless of which projection mode is used.
Here we have a different need, to convert things to pixels, so that the shader can do math in pixels. My assumption is that we need a new set of functions in the project package.

Something like

project_scale_to_meters
project_scale_to_pixels

But these can not both be passed to the projectionMatrix, so we need an additional project function in the middle.
As much as I'd love to have a PR on this, it will need very careful thought and review by the deck.gl team. If we really want to do this now, we'll need to take the time to get a proposal up for how we extend project package.

@ibgreen Would it be okay to just put in the fix I had in initially, where I simply convert radiusMaxPixels and radiusMinPixels into meters in the ScatterLayer shader? This would fix the bug I am concerned about.

@huaruiwu we plan to land your PR after we add some test cases for the meter offset mode. hopefully very soon

okay thank you!

With this PR applied, in the scatterplot (meters mode) rendering test, pixel match increased from 12.306% to 94.068%. The remaining difference is presumedly from treating pixels per meter as a uniform.

awesome ! thank you @Pessimistress


AGM-Map Is not showing

I am using AGM Map in my angular 6 application.when i run application locally it is working but same application when i published to the server it is not opening map

before logging a bug, ask usage questions on our discord


.NET Core 3.0 compatibility

Mentioned in #284 - .NET Core 3 will be supported in a new release.
Now that .NET Core 3 is released, when can we expect work starting towards a working release?

@aelij
AvalonEdit is now released in V6 wich has NetCore support

master is already compiled against Core 3.0 (see 221e4568ab0eee2a926ccd596e58b36a606b94a4).
I'm updating the build system to support running 3.0 code. As to when, I cannot say exactly :) Hopefully in the upcoming weeks.

really looking forward to this!
I hope the build-system will support wpf/forms aswell - got many snippets to test against Core :)

could you also switch to avalonedit V6?

@jogibear9988 Yeah, probably

.NET Core 3.1 is now supported in master (see 6cdf08505caeefdc5a04a1aeab0499a2726df684). Will be releasing version 15 soon. Note this version will require the .NET Core SDK to work (even to run .NET Framework scripts), since part of the compilation is now handled by MSBuild.

Running master raises an issue (probably because my username contains a space)

Thanks, @Kirides. Can you submit a PR? :)

FYI, the new version also supports .NET Core's "framework references". Here's how you can use WPF:
#r "framework:Microsoft.WindowsDesktop.App"

using System.Windows;
using System.Windows.Controls;

await Helpers.RunWpfAsync(); // initializes a dispatcher thread

var w = new Window { Content = new Button { Content = "Hello, WPF", IsCancel = true } };
w.ShowDialog()

someone pls help me icant do anything ((( roslynpad doesn't have docs for beginers  that is sad


Code Execution vulnerability with tool PythonCodeTool

Bug Description
When compose an LLM app with langflow, PythonCodeTool is available to developers to implement a tool with StructuredTool in langchain. However, there is a lack of validation for the python code and codes will be executed directly.
Once the LLM app is deployed on a server, arbitrary code can be executed on the server.
async def build(
        self,
        tool_code: str,
        name: str,
        description: str,
        tool_function: List[str],
        return_direct: bool,
        tool_class: Optional[List[str]] = None,
    ) -> Tool:
        local_namespace = {}  # type: ignore
        exec(tool_code, globals(), local_namespace)

Reproduction
1.Create a new project.
2.Import the json file to the collection.
PythonCodeTool.json
In the PythonCodeTool, Tool Code is set to read /etc/passwd with os.popen.
def search_function(query: str):
     import os
    return os.popen('cat /etc/passwd').read()

search = StructuredTool.from_function(
    func=search_function,
    name="Search",
    description="useful for when you need to answer questions about current events",
    # coroutine= ... <- you can specify an async method if desired as well
)

And in order to use the search tool,  the input is set to Search for the result.
Also input a valid OPENAI API KEY and OPENAI BASE.
3.Run Flow in the Playground. Code in PythonCodeTool is executed and we can get the content of /etc/passwd of the server.

Expected behavior
Since all the inputs can be controled by users, validation should be taken into consideration when executing users code. Enable the necessary action and forbid the malicious behavior before using exec.
Who can help?
No response
Operating System
Ubuntu Linux 22.04
Langflow Version
v1.0.12
Python Version
None

I share the same opinion. Using exec is very risky, and I was concerned about it while developing. I am currently working on a new design.
Preview
import ast
from typing import Any, Dict

from langchain.agents import Tool
from langflow.inputs.inputs import MultilineInput, MessageTextInput, BoolInput, DropdownInput
from langchain_core.tools import StructuredTool
from langflow.io import Output

from langflow.custom import Component
from langflow.schema.dotdict import dotdict

class PythonCodeStructuredTool(Component):
    display_name = "Python Code Tool"
    description = "structuredtool dataclass code to tool"
    documentation = "https://python.langchain.com/docs/modules/tools/custom_tools/#structuredtool-dataclass"
    name = "PythonCodeStructuredTool"
    icon = "🐍"
    field_order = ["name", "description", "tool_code", "return_direct", "tool_function", "tool_class"]
    inputs = [
        MultilineInput(
            name="tool_code",
            display_name="Tool Code",
            info="Enter the dataclass code.",
            placeholder="def my_function(args):\n    pass",
            refresh_button=True,
        ),
        MessageTextInput(name="tool_name", display_name="Tool Name", info="Enter the name of the tool."),
        MessageTextInput(name="tool_description", display_name="Description", info="Enter the description of the tool."),
        BoolInput(name="return_direct", display_name="Return Directly", info="Should the tool return the function output directly?"),
        DropdownInput(
            name="tool_function",
            display_name="Tool Function",
            info="Select the function for additional expressions.",
            options=[],
            refresh_button=True,
        ),
        DropdownInput(
            name="tool_class",
            display_name="Tool Class",
            info="Select the class for additional expressions.",
            options=[],
            required=False,
            refresh_button=True,
        ),
    ]
    outputs = [
        Output(display_name="Tool", name="result_tool", method="build_tool"),
    ]

    def parse_source_name(self, code: str) -> Dict:
        parsed_code = ast.parse(code)
        class_names = [node.name for node in parsed_code.body if isinstance(node, ast.ClassDef)]
        function_names = [node.name for node in parsed_code.body if isinstance(node, ast.FunctionDef)]
        return {"class": class_names, "function": function_names}

    def update_build_config(self, build_config: dotdict, field_value: Any, field_name: str | None = None) -> dotdict:
        if field_name == "tool_code" or field_name == "tool_function" or field_name == "tool_class":
            try:
                names = self.parse_source_name(build_config.tool_code.value)
                build_config["tool_function"]["options"] = names["function"]
                build_config["tool_class"]["options"] = names["class"]
            except Exception as e:
                self.status = f"Failed to extract class names: {str(e)}"
                build_config["tool_function"]["options"] = ["Failed to parse", str(e)]
                build_config["tool_class"]["options"] = ["Failed to parse", str(e)]
        return build_config
    def find_imports(self, code: str) -> dotdict:
        imports = []
        from_imports = []
        parsed_code = ast.parse(code)
        for node in parsed_code.body:
            if isinstance(node, ast.Import):
                for alias in node.names:
                    imports.append(alias.name)
            elif isinstance(node, ast.ImportFrom):
                from_imports.append(node)
        return {"imports": imports, "from_imports": from_imports}
    async def build_tool(self) -> Tool:
        local_namespace = {}  # type: ignore
        tool_code = f"from langchain_core.pydantic_v1 import BaseModel, Field\n{self.tool_code}"
        modules = self.find_imports(tool_code)
        import_code = ""
        for module in modules["imports"]:
            import_code += f"global {module}\nimport {module}\n"
        for from_module in modules["from_imports"]:
            for alias in from_module.names:
                import_code += f"global {alias.name}\n"
            import_code += f"from {from_module.module} import {', '.join([alias.name for alias in from_module.names])}\n"
        exec(import_code, globals())
        exec(tool_code, globals(), local_namespace)

        func = local_namespace[self.tool_function]
        _class = None

        if self.tool_class:
            _class = local_namespace[self.tool_class]

        tool = StructuredTool.from_function(
            func=func, args_schema=_class, name=self.tool_name, description=self.tool_description, return_direct=self.return_direct
        )
        return tool  # type: ignore

    def post_code_processing(self, new_frontend_node: dict, current_frontend_node: dict):
        """
        This function is called after the code validation is done.
        """
        frontend_node = super().post_code_processing(new_frontend_node, current_frontend_node)
        frontend_node["template"] = self.update_build_config(
            frontend_node["template"], frontend_node["template"]["tool_code"]["value"], "tool_code"
        )
        frontend_node = super().post_code_processing(new_frontend_node, current_frontend_node)
        return frontend_node

There are options in LangChain's StructuredTool, such as the option to extract signatures. I am considering a format that reuses such options and does not use the Code field. If you have any suggestions for improvement, I will consider them

To avoid the risk for exec, we can:

Limit imports after find_imports function in the below code. Such as os, subprocess and sub modules like langflow.utils.validate.importlib.resources.os.

Since there are some modules which cannot be avoided, try to execute codes with a docker environment.

Hi @ogabrielluiz,
What's the security policy of langflow? I wonder if maintainers will patch the vulnerability and request for a CVE.
Thank you.

I don't think any update on this component is worth it in terms of security. Even implementing a sandbox is not enough to actually prevent malicious users to access the system, there are too many ways to escape it.
Langflow admin must be aware of it and do not let any client to execute code.
Any component can be customized if you have access to the UI or the API to import a flow.
Langflow flows must be considered as application code, therefore it's not up to langflow runtime code to provide those mitigation. This can be achieved by Authorization implementation in the backend, which is lacking today but can "easily" replaced by any other http proxy with custom rules.

@0gur1
Do you need any assistance with this case? If not, please let us know if this issue can be closed.

https://github.com/langflow-ai/langflow/issues/1973
You can deploy Langflow in BE mode with no UI and then it is up to the Administrator/Operator to put API restrictions based on their deployment environment


Gendered Language: Feature or Bug in Software Documentation?

I was inspired by https://modelviewculture.com/pieces/gendered-language-feature-or-bug-in-software-documentation.
This PR adds tests to all docstrings in our library to prevent gendered language from leaking in

Also inspired by this https://modelviewculture.com/pieces/the-open-source-identity-crisis

For a practical review of ways to avoid gendered assumptions without resorting to contorted English, see Brian Garder, author of "Modern American Usage" in a series of three short posts. The most relevant is this one.  @ericdill maybe add a link to this PR?

Is full-on blacklisting the male pronoun is the right way to ensure openness in the library? Sometimes we might want to refer to the work of a specific researcher who has a gender. Maybe a warning would be better than a test failure.

:+1: on this.
I am inclined to say we deal with the case where we must use a pronoun to refer to a researcher in a docstring at a when it comes up.  I suspect that if we have enough text in the docstring that using a pronoun is required there is too much text and it should move off to the rst documents anyway.

It looks like there is some accidentally-committed code in here.

It is not accidentally committed.  This new test actually imports every module and apparently some of our relative imports were incorrect

@danielballan Added a link to the writing guidelines. Thanks for that

Also, this should really make use of regex, but I suck at it


Update version to 1.0.83-4 on staging

Update version to 1.0.83-4

🚀 Deployed to production by @isagoico in version: 1.0.85-9 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Does not work in Safari; does in Chrome

Hi,
Not sure why (tried a lot with CORS and code) but it isn't working in Safari.
It does (sometimes) work in Chrome but not always.
Do not get errors other than "Error:" as an alert with no content.
In apache error log; (103)Software caused connection abort, AH01075: Error dispatching request to : (reading input brigade)
Do you have any other clue where to look?

Hummm, can you try this?
Change the form label to a simply div label in lines 6 and 21 on public/login.html
I got same problem on iOS and it solve the problem for me.
Was weird because in Windows (with Chrome) all worked fine, but on iOS (Safari) had exactly that problem.

Wow; that was it indeed. Works instant now. Thanks! :)


Automerge support for Tekton 0.12

In testing the https://github.com/rhd-gitops-example/services/tree/master/automerge-example @dibbles noticed errors of the form
HEAD detached from d88df61
[do-this-first : merge-pr] nothing to commit, working tree clean

when using Tekton Pipelines 0.12. These errors are not seen with Pipeline 0.11.3. The PipelineResources in the automerge-example need to be updated and re-tested with Tekton 0.12+. The failure is believed to be due to an optional refspec option introduced in the 0.12 timeframe. See https://github.com/tektoncd/pipeline/commit/8bc424db6249a4f4ab9240792a4a6b1ec00b4222

revision: Git revision (branch, tag, commit SHA or ref) to clone. You can use this to control what commit or branch is used. git checkout is used to switch to the revision, and will result in a detached HEAD in most cases. Use refspec along with revision if you want to checkout a particular branch without a detached HEAD. If no revision is specified, the resource will default to master.

Fetch the develop branch and switch to it (not detached)
  revision: develop
  refspec: refs/heads/develop:refs/heads/develop

Ensure that both the 'standalone' and 'webhooks' PipelineResources are updated and tested.

Grabbing this as well for now to work in tandem with https://github.com/rhd-gitops-example/services/issues/55

I'll see if I can fix this for v1alpha1 on Tekton 0.12.x, independently of #101.


Unable to run example project

I'm having issues getting the basic generated sample to run. I have tried react-native-desktop before and got it working right away, but today I'm having issues.
I have run:
brew install watchman
npm install -g react-native-desktop-cli
react-native-desktop init ExampleProject
cd ExampleProject
npm start

I then open the XCode project and run the app. It builds fine, but when getting the Javascript it errors with:
2016-07-14 23:07:01.460 [fatal][tid:main] Unable to resolve module react-native/lib/InitializeJavaScriptAppEngine from <PROJECT_PATH>/node_modules/react/lib/ReactNativeDefaultInjection.js: Unable to find this module in its module map or any of the node_modules directories under /Users/node_modules/react-native/lib/InitializeJavaScriptAppEngine and its parent directories

This might be related to https://github.com/facebook/react-native/issues/4968
To resolve try the following:
  1. Clear watchman watches: `watchman watch-del-all`.
  2. Delete the `node_modules` folder: `rm -rf node_modules && npm install`.
  3. Reset packager cache: `rm -fr $TMPDIR/react-*` or `npm start -- --reset-cache`.

I have tried all the options it gives, but it seems the error is coming from line 17 of react/lib/NativeMethodsMixin.js where the react module is attempting to bridge into react-native instead of react-native-desktop. I have tried the solution from #87 where I create a dummy package called react-native which just exports react-native-desktop, however, this does not seem to work.
Any ideas what the solution might be?

Oops. React has added the direct dependency for react-native. Let me see that I can do about it.

Thanks! Do you know if there's a combination of versions of react and react-native-desktop which I can use in the meantime. I tried looking in the react source code for when they made the change, but I can't actually see any of the source code in their repo. It looks like they have a really strange build system structure which changes where all the code lives...

You could try:
rm -rf node_modules/react node_modules/react-native-desktop
npm install<EMAIL_ADDRESS>

Please try again to re-create a project or directly specify 0.9.0-rc1.

Thanks! That works brilliantly!


Wrong generated code when using import alias for Dart tables

Hello!
I noticed that when using an import alias for the tables property in the DriftDatabase annotation the generated code is not using the alias, so it won't compile.
I'm not sure how actionable this issue is, I'm sure it's not that common to do this. To give more context, in electricsql a drift schema is generated automatically based on a Postgres schema, so that then the user only needs to do @DriftDatabase(tables: kElectrifiedTables). The tables list is a constant that is also generated automatically ( List<Type>).
This works, but an user reported that the drift code gen wasn't working when using an import alias for the schema file.
It can be reproduced by editing the app example in the repo as follows:

Thanks for opening the issue, I actually wanted to fix this for a long time! In the writer, we only get a name of the element and its definition uri and we have to resolve its import alias based on that. It's not a fully sound solution, but most instances of this should be fixed by 261c151fbf87a580b07bc8565afbd9d7f16e9a07. That looks up all named imports in a library and picks the first one that matches. It's not correct because, if a name is imported both with and without a name, we won't always pick the right one now. But it's likely that most of these cases are already broken today, and this will at least improve some of them.

Thanks Simon, as always! 😄


authorization based on claims and claims values

Added claims parameter in the decorator and check for required claims and optionally specific claim value.
We dont need to have that, since it can be done in the endpoint after our filter has run, so feel free to decline this PR if you don't find it useful

Agree. Do that in another PR when applicable