Cross compilation from x86_64 to x86_64 with custom gcc arch/tune fails

Steps to reproduce
I have small VPS which could benefit greatly from AES-NI and other fancy instructions. It can't be used for building due to load limitations, so I've attempted to do compilation on local machine and push that on remote target:
nixos-rebuild --flake .#myServer \
  --target-host servername \
  --build-host localhost \
  boot

Wiki suggest tweaking configuration for such scenario in this way:
nixpkgs = {
    # building ON
    localSystem = {
        system = "x86_64-linux";
    };
    # building FOR
    crossSystem = {
        gcc.arch = "znver2";
        gcc.tune = "znver2";
        system = "x86_64-linux";
    };
};

However it fails while trying to compile bash:
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu

Beginning configuration for bash-5.2-release for x86_64-unknown-linux-gnu

checking for x86_64-unknown-linux-gnu-gcc... x86_64-unknown-linux-gnu-gcc
checking whether the C compiler works... no
configure: error: in `/build/bash-5.2':
configure: error: C compiler cannot create executables

After that I've tried alternative approach that doesn't touch localSystem / crossSystem and modifies compiler wrapper instead:
crossOverlays = [
    (final: prev: {
        wrapCC = cc: prev.wrapCCWith {
            inherit cc;
            extraBuildCommands = ''
                echo "-O3 -march=znver2 -mtune=znver2" >> "$out/nix-support/cc-cflags"
            '';
        };
    })
];

Sadly that failed on python (bash worked though):
Found duplicated packages in closure for dependency 'setuptools':
        setuptools 68.2.2.post0 (/nix/store/rk7kwr8c0v8dwk64442q8c0lp7car7qn-python3.11...
        setuptools 68.2.2.post0 (/nix/store/y8yq448xka665bqyd5gvdhajqdbsnyys-python3.11...

Notify maintainers
I've seen @amjoseph-nixpkgs working on cross compilation a lot, I assume it's his area of expertise
Metadata
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.62, NixOS, 23.11 (Tapir), 23.11.20231117.c757e9b`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `""`
 - nixpkgs: `/nix/store/rj6pqzx66pliiwdfk2wjnnlk40rm3mpd-source`

Dupe of https://github.com/NixOS/nixpkgs/issues/243164
localSystem should work
https://github.com/search?q=%2Fgcc\.tune%2F+lang%3Anix+NOT+is%3Afork&type=code
There's a simpler way to add flags than what you're doing
https://github.com/NixOS/nixpkgs/blob/4ccb3e083b4d83964897efd1c2728b7c33b47239/pkgs/stdenv/adapters.nix#L287
https://github.com/search?q=withCFlags+lang%3Anix+NOT+is%3Afork&type=code

@Artturin stdenv adapters stuff seems to be broken. Doing it this way in latest nixos-unstable (global overlay):
self: super: {
    stdenv = super.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] super.stdenv;
}

always results in error
error: The ‘env’ attribute set cannot contain any attributes passed to derivation. The following attributes are overlapping: NIX_CFLAGS_COMPILE`

I've tried other adapters mentioned here, it's all the same
As for localSystem, I think last time I tried it just built optimized packages locally, but normal packages were copied to target. Same goes for overlays (that's why crossOverlays). It could be that I simply forgot to reboot though, I am going to retest.
But anyways, man  nixos-rebuild has such line:
--target-host host
...
Note that nixos-rebuild honors the nixpkgs.crossSystem setting of the given configuration but disregards the  true  architecture  of  the  target host. Hence the nixpkgs.crossSystem setting has to match the target platform or else activation will fail.

It implies that crossSystem should be used, not localSystem.

@Artturin stdenv adapters stuff seems to be broken. Doing it this way in latest nixos-unstable (global overlay):
self: super: {
    stdenv = super.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] super.stdenv;
}

always results in error
error: The ‘env’ attribute set cannot contain any attributes passed to derivation. The following attributes are overlapping: NIX_CFLAGS_COMPILE`

This is because something is setting NIX_CFLAGS_COMPILE instead of env.NIX_CFLAGS_COMPILE, it would probably be possible to choose which one to use with some conditionals, but I'm not inclined to do it because people should switch to using the env attrset for environmental variables within builds to support structuredAttrs.

Answer before realizing the error was about overlapping attrs
Works fine here
nix build --expr '(import ./. { overlays = [(self: super: {stdenv = super.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] super.stdenv;  })]; }).zsh' --impure --keep-failed

because toString is run on it https://github.com/NixOS/nixpkgs/blob/3ccf65f738553bf3d3ee7ebeb2e885c2d9653007/pkgs/stdenv/adapters.nix#L290
however it fails with stuff like
expand-response-params> cc1: error: bad value ('znver2') for '-march=' switch
expand-response-params> cc1: note: valid arguments to '-march=' switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client icelake-server bonnell atom silvermont slm knl knm x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 btver1 btver2 native; did you mean 'znver1'?

(likely related https://github.com/NixOS/nixpkgs/issues/253713)
To avoid the bootstrapping stuff do  import ./. { config = { replaceStdenv = ({ pkgs }: pkgs.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] pkgs.stdenv); }; };
This still fails with errors like
> g++ -g -O2  -o gperf version.o positions.o options.o keyword.o keyword-list.o input.o bool-array.o hash-table.o search.o output.o main.o ../lib/libgp.a -lm
> /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: /build/ccPAfibH.ltrans0.ltrans.o: in function `Input::read_input()':
> /build/gperf-3.1/src/./input.cc:254: undefined reference to `get_delim(char**, unsigned long*, int, _IO_FILE*)'
> /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: /build/ccPAfibH.ltrans0.ltrans.o: in function `Hash_Table::insert(KeywordExt*)':
> /build/gperf-3.1/src/./hash-table.cc:143: undefined reference to `hashpjw(unsigned char const*, unsigned int)'


sort with complex {...} doesn't work

sort with complex {...} doesn't work.  For example (from CGI/Util.pm):
foreach my $k (
    # sort keys alphabetically but favour certain keys before others
    # specifically for the case where there could be several options
    # for a param key, but one should be preferred (see GH #155)
    sort {
        if    ( $a =~ /content/i ) { return 1 }
        elsif ( $b =~ /content/i ) { return -1 }
        else  { $a cmp $b }
    }
    keys( %params_as_hash )
) {...}

gives:
ATTENTION!
The following lines were probably translated incorrectly
on the 'if' statement.

Fixed in v1.002


Team page js

I had to remove turbolinks, don't know if anyone cares about that.
But now it's really easy to just copy and paste new dots into the team page and the rows get formatted nicely.
The tooltips show up in funny places if there are only 1 or 2 dots in a row.  I think I can try to fix that later, but I'll do it in another branch.

Looks great, thanks @brigid2718  :)

@brigid2718 And I think we can think about Turbolinks or something similar at a later stage again...


[BUG] Timeliness monitor fails under repeated start/stop

Version information

RDK: v0.8
Robot software: v2.10.5
OS: Ubuntu 20.04 - x86_64

Describe the bug
Timeliness monitor cut off the RDK connection under repeated mode switching.
Steps to reproduce
Compile and run the following minimum reproducible example (make sure your robot arm is not in a singularity, and modify the hard coded robot IP and local IP):
#include <flexiv/Robot.hpp>
#include <flexiv/Exception.hpp>
#include <flexiv/Log.hpp>
#include <flexiv/Scheduler.hpp>
#include <flexiv/Utility.hpp>
#include <flexiv/Model.hpp>
#include <thread>
#include <iostream>

void periodicTask(flexiv::Robot& robot, flexiv::Scheduler& scheduler, flexiv::Log& log,
                  const std::vector<double>& initTcpPose)
{
  static int i = 0;
  static int N = 1000; // It works if N = 5000(5s)
  if (i % N == 0)
    robot.setMode(flexiv::Mode::RT_CARTESIAN_MOTION_FORCE_BASE);
  else if(i % N > 0 && i % N < N-1)
    robot.streamCartesianMotionForce(initTcpPose);
  else
    robot.stop();

  i++;

  if (i>10000)
  {
    robot.stop();
    scheduler.stop();
  }
}

int main(int argc, char* argv[])
{

  // Program Setup
  // =============================================================================================
  // Logger for printing message with timestamp and coloring
  flexiv::Log log;

  // IP of the robot server
  std::string robotIP = "<IP_ADDRESS>";
  // IP of the workstation C running this program
  std::string localIP = "<IP_ADDRESS>";

  try {
    // RDK Initialization
    // =========================================================================================
    // Instantiate robot interface
    flexiv::Robot robot(robotIP, localIP);

    // Create data struct for storing robot states
    flexiv::RobotStates robotStates;

    // Clear fault on robot server if any
    if (robot.isFault()) {
      log.warn("Fault occurred on robot server, trying to clear ...");
      // Try to clear the fault
      robot.clearFault();
      std::this_thread::sleep_for(std::chrono::seconds(2));
      // Check again
      if (robot.isFault()) {
        log.error("Fault cannot be cleared, exiting ...");
        return 1;
      }
      log.info("Fault on robot server is cleared");
    }

    // Enable the robot, make sure the E-stop is released before enabling
    log.info("Enabling robot ...");
    robot.enable();

    // Wait for the robot to become operational
    int secondsWaited = 0;
    while (!robot.isOperational()) {
      std::this_thread::sleep_for(std::chrono::seconds(1));
      if (++secondsWaited == 10) {
        log.warn(
              "Still waiting for robot to become operational, please check that the robot 1) "
              "has no fault, 2) is in [Auto (remote)] mode");
      }
    }
    log.info("Robot is now operational");

    //Test the scheduler
    // Set initial TCP pose
    robot.getRobotStates(robotStates);
    auto initTcpPose = robotStates.tcpPose;
    log.info("Initial TCP pose set to [position 3x1, rotation (quaternion) 4x1]: "
             + flexiv::utility::vec2Str(initTcpPose));

    flexiv::Scheduler scheduler;
    scheduler.addTask(std::bind(periodicTask, std::ref(robot),
                                std::ref(scheduler), std::ref(log),
                                std::ref(initTcpPose)),
                      "HP", 1, scheduler.maxPriority());
    scheduler.start();

  }
  catch (const flexiv::Exception& e)
  {
    return 1;
  }
  return 0;
}

The above code does the following:

Switch to RT_CARTESIAN_MOTION_FORCE_BASE mode.
Stream command normally for 1 second. (if changed to execute 5 seconds, then no issue)
Stop robot, and goto step 1.

Expected behavior
The program work as usual.
Screenshots
[2023-10-11 13:54:59.296] [info] ================================================
[2023-10-11 13:54:59.296] [info] ===              Flexiv RDK 0.8              ===
[2023-10-11 13:54:59.296] [info] ================================================
[2023-10-11 13:54:59.297] [info] [flexiv::Robot] Network service started [RT:99]
[2023-10-11 13:54:59.797] [info] [flexiv::Robot] Connection established
[2023-10-11 13:54:59.799] [info] [flexiv::Robot] Server information:
[2023-10-11 13:54:59.799] [info] Serial number: Rizon 4s-062060
[2023-10-11 13:54:59.799] [info] Degrees of freedom: 7
[2023-10-11 13:54:59.799] [info] Software version: v2.10.5
[2023-10-11 13:54:59.802] [info] [flexiv::Robot] Initialization complete
[2023-10-11 13:54:59.802] [info] Enabling robot ...
[2023-10-11 13:54:59.827] [info] Robot is now operational
[2023-10-11 13:54:59.827] [info] Initial TCP pose set to [position 3x1, rotation (quaternion) 4x1]: 0.651 -0.165 0.461 0.091 0.022 0.995 0.030
[2023-10-11 13:54:59.842] [info] [flexiv::Scheduler] Initialization complete
[2023-10-11 13:54:59.842] [info] [flexiv::Scheduler] Maximum available priority for the user task is [97]
[2023-10-11 13:54:59.842] [info] [flexiv::Scheduler] Added task [HP]: interval [1 ms], priority [97], CPU affinity [-1]
[2023-10-11 13:54:59.842] [info] [flexiv::Scheduler] Starting all tasks
[2023-10-11 13:55:01.847] [info] [flexiv::Robot] Operation mode transited to [RT_CARTESIAN_MOTION_FORCE_BASE]
[2023-10-11 13:55:02.853] [info] [flexiv::Robot] Operation mode transited to [IDLE]
[2023-10-11 13:55:03.693] [warning] [Timeliness Monitor] 10 missed messages in 5s
[2023-10-11 13:55:03.702] [warning] [Timeliness Monitor] 20 missed messages in 5s, RDK connection is about to close

Additional context
NA

And by modifying the periodicTask to
void periodicTask(flexiv::Robot& robot, flexiv::Scheduler& scheduler, flexiv::Log& log,
                  const std::vector<double>& initTcpPose)
{
  static int i = 0;
  static int N = 8000; // It works if N = 5000(5s)
  if (i % N == 0)
  {
    log.info("Start");
    robot.setMode(flexiv::Mode::RT_CARTESIAN_MOTION_FORCE_BASE);
  }
  else if(i % N > 0 && i % N < N-1)
  {
    if (i%100 == 0)
      log.info("Stream " + std::to_string(i));

    robot.streamCartesianMotionForce(initTcpPose);
  }
  else
  {
    log.info("-----------STOP------------");
    robot.stop();
  }

  i++;

  if (i>10000)
  {
    robot.stop();
    scheduler.stop();
  }
}

We noticed that for the several hundreds of cycles of periodicTask after the robot.setMode(flexiv::Mode::RT_CARTESIAN_MOTION_FORCE_BASE); call, the actual frequency is no-longer at 1000Hz, but at a frequency much higher than 1000 Hz.

@acf986 I modified your example to make sure the commands are streamed for 1s instead of 0.999s in your original example, but I still observed the timeliness monitor error. If I increased the N to 2000 (2s), then no issue. @pzhu-flexiv will check on this issue.
void periodicTask(flexiv::Robot& robot, flexiv::Scheduler& scheduler, flexiv::Log& log,
    const std::vector<double>& initTcpPose)
{
    static int i = 0;
    static int N = 1000;
    if (i % (N + 2) == 0) {
        robot.setMode(flexiv::Mode::RT_CARTESIAN_MOTION_FORCE_BASE);
    } else if (i  % (N + 2) == (N + 1)) {
        robot.stop();
    } else {
        robot.streamCartesianMotionForce(initTcpPose);
    }

    i++;

    if (i > 10000) {
        robot.stop();
        scheduler.stop();
    }
}

@munseng-flexiv May I know is your testing computer patched with a realtime kernel?

@munseng-flexiv May I know is your testing computer patched with a realtime kernel?

My workstation is Ubuntu 20.04 and is not patched with the real-time kernel. It might be the reason for the bug due to less stable whole-loop latency.

@acf986 I can't replicate the issue on RDK v0.9 with a real robot running robot software v2.10.7. I think the code cleanup and logic robustness improvements in RDK v0.9 has already fixed this bug.

May I know when will v0.9 be formally released?

May I know when will v0.9 be formally released?

The software package is at the last stage of QC, it should be available by the end of next week.

Fixed with release #40.


Fix unzipping 'lib' directory

This used to work when tested on Windows, but now was failing on macOS. Differences in Powershell versions?...

Thanks! We could probably set up a CI step that runs the script to test...


[feature\token-experiment] update control colors and a few style updates

Updating controls to use the new design token colors and a few style updates related to #3549 and #3546 in the feature\token-experiment branch. Given that this is mostly a find replace, some of the control colors might be off. There will be a follow up PR to fix up some of the specific instances where the colors are not correct.

/azp run

An initial look at the RadioButton Checked transitions, it feels a little slow.
I know this is early work but I thought I would mention it.
 <!-- 1.167 is relative scale from 12px to 14px -->
<SplineDoubleKeyFrame KeyTime="0:0:0.25" KeySpline="0.0,0.0 0.0,1.0" Value="1.167" />

I notice the Brush transition seems to use a much shorter duration KeyTime="0:0:0.083" so maybe something between the two?  I think it is mostly the PointerOver state that feels slow

Thanks for the feedback @mdtauk
@YuliKl @karenbtlai @stmoy as FYI.


K046 and K014 missing metadata

K014 has no metadata for wetland
K046 transition soil may be mislabeled

@kkhomolka / @opalotenburg Scavenger hunt to find the jars, bags and hyprop rings for K046 & K014 (unused jars from the kit). Cross check with kit level metadata for if these samples should exist or not.

Kit level metadata: https://drive.google.com/file/d/1Oeu5i_qCgj-AG9MlfbDn9jAkjfWUqtOS/view?usp=sharing

do you want me to put the info in that metadata sheet above? I found the empty jars/hyprops in our storage area and I just recorded this in a column to this sheet at the far right: [https://docs.google.com/spreadsheets/d/19F1oS-DBvxQlU1EYXtciYkhu6TNA7fuCQ0Ts2NRWQlk/edit#gid=1675812649] let me know if I should move this- wasn't sure where to put it in the other metadata sheet. For K014 I'm confused because I only found empty transition soil jar, hyprop, and bag. But for upland and wetland I only found the hyprops. Not sure if these were actually sampled or not. For K046 I was only able to find empty transition hyprop and wetland jar but it says in the metadata sheet that wetland soil was collected? I also didn't find the transition jar or bag so not sure if either of those were collected. @kkhomolka I think we need to crosscheck this against a soil sample inventory do you have one?

THANK YOU @opalotenburg and @kkhomolka, going to close this issue now

K046 jars were switched for transition and wetland, we will resolve this with #60
K014 was a metadata error: samples were collected in the wetland but did not record metadata for them


Apache Cordova

I have an application on polymer plataform , but for operational reasons, need to run plenty cellular and tables telephones , for which migrate over cordova apache, the problem that arises is that when I run emulating a cell the compononente not work for me without shed any error.
If I run the same application without touching any code emulating a browser works perfectly , you could guide me to solve the problem in advance thank you very much .

I'd be more than happy to help you out!
Might be best to exchange ideas in the Gitter chat

Before starting thank you very much for your attention , I happened to comment on apache cordova have problems , but not on the browser that works perfectly as you comment .

The filter and sort not work for me .
If the title frame check mark me all this is ok , but not brand me individually

Again many thanks for your time
Attach the file
nsc-msgview.zip

Hi!
I'm not entirely sure I understand what the problem is, but I added a simple example here how to do filtering with an external filter input: http://jsbin.com/facabu/edit?html,output
For the multi-selection, it works only on Chrome currently, there's a bug issued here: #63

@nscarg did you get everything working or is there still something?


Spring Cloud Gateway returning 400 on WebFlux Controllers

I have two Spring services running. One that is using normal spring web mvc and one that is using spring WebFlux. When I set up the following in my spring cloud gateway service:
 - id: orchestration-service
    uri: ${workflowApi.url:http://localhost:8000}
    predicates:
       - Path=/camunda/**

This works as expected as this is not a WebFlux controller. However the following:
- id: file-upload-service
  uri: ${fileUploadApi.url:http://localhost:8001}
  predicates:
      - Path=/files/**

Gives me a 400 (this service is using Spring WebFlux).  When I look at the logs the proxy is working correctly however I get this in the logs:
2021-06-03 16:36:14.621 TRACE [platform-gateway,,] 8576 --- [ctor-http-nio-5] o.s.c.gateway.filter.NettyRoutingFilter  : outbound route: 22b9014f, inbound: [f0ee0b98-1, L:/0:0:0:0:0:0:0:1:3333 - R:/0:0:0:0:0:0:0:1:59752]
2021-06-03 16:36:14.668 TRACE [platform-gateway,,] 8576 --- [ctor-http-nio-5] o.s.c.g.filter.NettyWriteResponseFilter  : NettyWriteResponseFilter start inbound: 22b9014f, outbound: [f0ee0b98-1, L:/0:0:0:0:0:0:0:1:3333 - R:/0:0:0:0:0:0:0:1:59752]
2021-06-03 16:36:14.679 TRACE [platform-gateway,,] 8576 --- [ctor-http-nio-5] o.s.c.g.filter.GatewayMetricsFilter      : spring.cloud.gateway.requests tags: [tag(httpMethod=GET),tag(httpStatusCode=400),tag(outcome=CLIENT_ERROR),tag(routeId=history-service),tag(routeUri=http://localhost:8003),tag(status=BAD_REQUEST)]

If I hit the file service url directly the API works correctly and returns results, but when I hit the endpoint via the gateway I get a 400. Is there anything I can do to debug further? Or am I missing a specific dependency or config?
Any help would be appreciated.
I'm using the following dependencies:
implementation 'org.springframework.cloud:spring-cloud-starter-gateway:3.0.3'

Can you provide a complete, minimal, verifiable sample that reproduces the problem?  It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

Hi @spencergibb I resolved the issue. It was a config issue that was causing the 400.  Sorry for the inconvenience!

I seem to have the same issue, could you provide more infos on which config item was problematic?


Channels support

I had a play around and managed to get this working at a basic level with the upcoming Channels support: https://github.com/strawberry-graphql/strawberry/pull/1407
It would be great for this package to support channels. From what I understand it requires some changes to support the difference context object.

In strawberry django auth I have tests for channels subscriptions.


Gracefully handle external network call failures

When Tola makes requests calls to external services like the DIG server during DIG indicator creation, or when creating a new project, it's possible these external API calls can fail. Right now the app just throws an exception.
Handle this failure case in such a way the end user is notified, and the app does not crash.

The existing code that makes external calls doesn’t seem to assume failure at all, and in the case of things like AJAX calls, doesn’t have an existing channel in place for reporting server side errors. I’m wondering if we should evaluate where reporting may be most needed as to not spend a bunch of time on reporting that is of little to no value to end users

All of the places in the codebase where requests are used:
            configurabledashboard
                views.py

                    Called once in dead code - IGNORE

            customdashboard
                views.py

                    Called 2 times

                    SurveyPublicDashboard
                      "http://tola-tables.mercycorps.org/api/silo/430/data/"

                    SurveyTalkPublicDashboard
                      "http://tables.toladata.io/api/silo/9/data/"

                    Pretty sure these are dead code paths - IGNORE

            indicators/views
                views_indicators.py

                    import_indicator()  # DIG indicator import function
                      pulls URL from ExternalService DB entry
                      Called by
                        indicator_create()  # when creating DIG indicator
                        service_json()  # populates dropdown box on indicator create

                	  I already fixed this code path to log on error

                	getTableCount()  # get number of rows in tola table
                	  CollectedDataCreate()
                	  CollectedDataUpdate()
                	  collecteddata_import()  # import collected data from tola tables

            scripts
                get_github_project_issues.py

                    This is a dev script - devs problem

            tables
                views.py

                    import_table() # import collected data from Tola Tables

                      pulls url from: ExternalService.objects.get(name="TolaTables")
                        this entry doesn't exist in my DB dump
                        this makes me think it's dead code!

                        IGNORE

            tola
                util.py

                    get_table()
                      # Get table data from a Silo.  First get the Data url from the silo details
                      Called by:
                        customdashboard/views PublicDashboard

                        views_indicators collecteddata_import()  # tola table data ??

            workflow
                views.py

                    import_service()  # yet another DIG indicator import function
                      called only by service_json()
                        this is in turn called by some JS which my or may not be dead

            workflow/management/commands
                fetch_gait_dates.py

                    dev's problem - IGNORE

I have deleted workflow.views.import_service()/service_json() as this was just copy n pasted from the indicators version (which is already reporting errors in logging).
Out of the list above, the only things that need error handling are:
tola.util.get_table()
tola.utils.get_GAIT_data()
indicators.views.views_indicators.getTableCount()
Two of these relate to TolaTables import which is know nothing about and could use some input on wrt how to handle error reporting.
The third is for pulling info from GAIT during program creation. For this, should Program creation continue without populating the fields normally populated by GAIT, or stop creation and give an error to the admin?

Since TolaTable importing from the collect data form is going away in Huckleberry, no work will be done for those.
That just leaves handling network errors during program creation which will still allow Program creation with error reporting if possible.

Closing as fixed and deployed.


Don't Merge! PR real time test for webhook case

Docs Build status updates of commit e6f6fb3:
:clock10: Pending: waiting for processors (8 builds ahead of you)
:warning: Docs Build is busy, currently there are 8 builds ahead of this one, for more information you can view the Build queue graph on the Docs Portal.

Docs Build status updates of commit e6f6fb3:
:clock10: Full build: the average full build time is 5 min(s) 20 sec(s), this is based on the last 30 build(s) for this repository.

Docs Build status updates of commit e6f6fb3:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs-ref-conceptual/testing.md
:warning:Warning
View (azure-cli-latest) View (azure-cli-2017-03-09-profile) View (azure-cli-2018-03-01-hybrid)
Details

docs-ref-conceptual/testing.md

[Warning] Missing attribute: either ms.prod or ms.service is required. Use ms.prod for on-premise products, or ms.service for cloud services.
[Warning] Missing attribute: title.
[Warning] Missing attribute: either ms.prod or ms.service is required. Use ms.prod for on-premise products, or ms.service for cloud services.
[Warning] Missing attribute: title.
[Warning] Missing attribute: either ms.prod or ms.service is required. Use ms.prod for on-premise products, or ms.service for cloud services.
[Warning] Missing attribute: title.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


TOCO fails to handle Dilated Convolution properly

System information
Have I written custom code (as opposed to using a stock example script provided in TensorFlow):N/A
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:N/A
TensorFlow installed from (source or binary):source
TensorFlow version (use command below):1.10.0
Python version: 2.7.12
Bazel version (if compiling from source):0.15.0
GCC/Compiler version (if compiling from source):5.4.0
CUDA/cuDNN version:N/A
GPU model and memory:N/A
Exact command to reproduce:N/A
Problem description
I have frozen_model.pb which works well.
I build toco with command:
bazel build -c opt //tensorflow/contrib/lite/toco:toco
Then i'm trying to convert model to TfLite:
toco --allow_custom_ops --input_file='frozen_model.pb'  --output_file='wavenet.tflite' --input_arrays='original_input' --input_shapes='1,48000,1' --output_arrays='activation_2/Softmax' --dump_graphviz=graph --dump_graphviz_video=true
and geting error:
2018-09-07 17:45:53.870648: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 412 operators, 674 arrays (0 quantized)
2018-09-07 17:45:53.876846: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 412 operators, 674 arrays (0 quantized)
2018-09-07 17:45:53.877237: I tensorflow/contrib/lite/toco/graph_transformations/identify_dilated_conv.cc:161] Identified sub-network emulating dilated convolution.
2018-09-07 17:45:53.877266: I tensorflow/contrib/lite/toco/graph_transformations/identify_dilated_conv.cc:209] Replaced with Dilated Conv2D op outputting "dilated_conv_2_tanh/convolution/Conv2D".
2018-09-07 17:45:53.877487: I tensorflow/contrib/lite/toco/graph_transformations/identify_dilated_conv.cc:161] Identified sub-network emulating dilated convolution.
2018-09-07 17:45:53.877512: I tensorflow/contrib/lite/toco/graph_transformations/identify_dilated_conv.cc:209] Replaced with Dilated Conv2D op outputting "dilated_conv_2_sigm/convolution/Conv2D".
2018-09-07 17:45:53.877852: F tensorflow/contrib/lite/toco/graph_transformations/propagate_fixed_sizes.cc:116] Check failed: dim_x == dim_y (48002 vs. 48000)Dimensions must match

This is Dilated Convolution block in original frozen_model.pb visualized in tensorboard, it has input and output shape [?, 48000, 40]:
[?, 48000, 40] ->Pad->[?, 48002, 40]->SpaceToBatchND -> ExpandDims -> Conv2D -> Squeeze -> BatchToSpaceND->[?, 48000, 40]

TOCO tries to substitute with
[?, 48000, 40] ->Pad->[?, 48002, 40]->Reshape -> Conv1x1/S -> Squeze->[?, 48002, 40]

so ouput of resulted block is [?, 48002, 40] instead of [?, 48000, 40], it causes error in subsequent operations.
I managed to avoid this issue by removing line 102:
transformations->Add(new IdentifyDilatedConv);
from tensorflow/contrib/lite/toco/toco_tooling.cc and rebuilding TOCO,
but unfortunately this causes new problems, because some operations like SpaceToBatchND and BatchToSpaceND don't support 3D tensors in current TfLite verison.

Thanks for the report, would it be possible to share the model (or truncated model) which yields this error? Thanks.

Here is simplified model created in Keras 2.2.2:
keras-dilated-conv.tar.gz

images for converted graph are also in archive

Nagging Assignee @robieta: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hello! What's the status of this issue? It seems like I have faced this bug too.

Nagging Assignee @suharshs: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @suharshs: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @suharshs: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

This error always oppears when you try to convert Keras model with Conv1D when dilation_rate > 1 and padding  != 'same'
Still on TF 1.13

@gashmish : Recently i uploaded 2 PRs to address SpaceToBatchND and BatchToSpaceND arbitrary dimension support, i have verified with the model you uploaded in this issue, it works fine. Would you please consider these 2 PRs into your workspace and check with your real time data once. Just need to know if it solves your blocking issue, Thanks!
2 PRs are #27867 & #28179

Hello @ANSHUMAN87 !
I tested your PRs for our model in TF Lite and it works well, great job!
Regarding current issue, if those PRs are applied, some option for disabling ResolveDilatedConv transflormation in TOCO could solve the problem
However, strainforward disabling ResolveDilatedConv may cause some other errors in subsequent graph nodes, like this:
ERROR: tensorflow/lite/kernels/strided_slice.cc:152 op_context.input->type != op_context.output->type (2 != 1) ERROR: Node number 216 (STRIDED_SLICE) failed to prepare.
I managed to avoid this by removing Reshape layer in Keras model, but in general there can be other problems

@gashmish : Thanks a lot for confirming!
I am unsure why you need to disable ResolveDilatedConv().
As i can reproduce the issue:

NumDimensions(op_context.input) != kInputDimensionNum (3 != 4)Node number 1 (SPACE_TO_BATCH_ND) failed to prepare.

without any change in code with the model you shared in zip file earlier.
Then i applied my 2 PRs, and the inference is successful.
Please let me know if i miss anything.
TIA!

@ANSHUMAN87, may be it was other model accidentally? By default TOCO tries to substitute dilated convolution subgraph with subgraph without SpaceToBatch and BatchToSpace
Converting model.pb to model.tflite from attached zip in TF nightly build with command
toco \
    --graph_def_file='model.pb' \
    --output_file="model.tflite" \
    --input_arrays='input' \
    --input_shapes='1,48000,40' \
    --output_arrays='dilated_conv_2/Tanh' \
    --dump_graphviz_video \
    --dump_graphviz_dir='graph'

toco_AT_IMPORT.dot.pdf
toco_AFTER_TRANSFORMATIONS.dot.pdf
In latest TF version you can use this script to create sample model:
import tensorflow as tf
from keras        import backend as K
from keras.layers import Input, Conv1D
from keras.models import Model

K.set_learning_phase(False)

input  = Input(
    name =  'input',
    shape = (48000, 40))

output = Conv1D(
    name          ='dilated_conv_2',
    filters       = 40,
    kernel_size   = 2,
    dilation_rate = 2,
    padding       = 'causal',
    activation    = 'tanh'
)(input)

model = Model(input, output)
model.summary()
model.save("sample.h5")

converter = tf.lite.TFLiteConverter.from_keras_model_file("sample.h5")
tflite_model = converter.convert()
open("sample.tflite", "wb").write(tflite_model)

Point of current issue is that subgraph substitution in this case works inproperly, so i needed to disable it in TOCO source

@gashmish : I have used tensorflow 1.13 for your model conversion to tflite with the script you shared. It executed perfectly fine. Also i went through the *.dot files you shared, the conversion is perfectly fine. The output shape is supposed to be [?, 48002, 40] as per padding. So still i am not sure where you are facing the issue. Would you please try converting your model using 1.13 version or later, and post your results.

@ANSHUMAN87: Everything i all right, by "this issue" i mean current topic about TOCO converter.

In TF 1.13.1 if I create and convert simple model with above script I get model with Conv2d and without BatchToSpace and SpaceToBatch. Output size is  [?, 48002, 40] instead of  [?, 48000, 40], so my production model will not work. This is ResolveDilatedConv transformation Bug.

If I disable ResolveDilatedConv transformation in toco I get model with BatchToSpace and SpaceToBatch. Then I apply your patches to TF Lite and everything works perfect.

Script and resulting models
So my problem is solved, but I still need to disable ResolveDilatedConv in TOCO source by hand, which also caused STRIDED_SLICE problem, which i avoided by also slightly changing or production model.
Regarding your BatchToSpace and SpaceToBatch ops changes everything works perfectly with our production model.

@ANSHUMAN87:

The output shape is supposed to be [?, 48002, 40] as per padding

In original Keras model and in Tensorflow model we get 48000 when using padding='causal'
Layer (type)                 Output Shape              Param #
=================================================================
input (InputLayer)           (None, 48000, 40)         0
_________________________________________________________________
dilated_conv_2 (Conv1D)      (None, 48000, 40)         3240
=================================================================

@gashmish : I am extremely sorry, i did some misinterpretation of the issue. Now i got the issue clearly.
There is a missed scenario which need to be handled in ResolveDilatedConv(). I will upload a fix for this and update you on that. Thanks a lot for your detailed reply.
Also i am little curious about your STRIDED_SLICE problem. Because no matter you disable as many Transformation like ResolveDilatedConv(), it should not break the Graph in the manner that subsequent Subgraphs will get impacted. If it is okay for you, would you be able to share small piece of code where this scenario is reproducible, i have tried with the steps you mentioned(like disable ResolveDilatedConv(), upload 2 PRs i shared), but did not reproduce.

@gashmish : I have uploaded fix for your problem, please refer #28410. I have verified. Request you also to verify with this change. And do let me know your results.
Also request you to share the code snippet or steps to reproduce STRIDED_SLICE problem. I suspect this could be some other issue.
Thanks a lot!

@ANSHUMAN87 : thanks for the PR. Hopefully with your change, we'll be able to convert the WaveRNN model through TOCO.
@gashmish: like Jared asked, may i ask for a direct pointer to where you got the WaveRNN model from? (e.g. Tensorflow Model Zoo, Github, etc ..). This is an important enough model that there should be conversion test to ensure conversion support stays going forward. If not, it's ok, I can try to find a canonical model online

@ANSHUMAN87: I tried to test PR #28410, unfortunately it didn't work for me.
I use this script:
model_name = 'sample'
model_h5     = model_name + '.h5'
model_tflite = model_name + '.tflite'

# 1. Create

K.set_learning_phase(False)

input  = Input(
    name =  'input',
    shape = (48000, 40))

output = Conv1D(
    name          ='dilated_conv_2',
    filters       = 40,
    kernel_size   = 2,
    dilation_rate = 2,
    padding       = 'causal',
    activation    = 'tanh'
)(input)

model = Model(input, output)
model.summary()
model.save(model_h5)

# 2. Convert

converter = tf.lite.TFLiteConverter.from_keras_model_file(model_h5)
tflite_model = converter.convert()
open(model_tflite, "wb").write(tflite_model)

# 3. Test

interpreter = tf.lite.Interpreter(model_path=(model_tflite))
interpreter.allocate_tensors()

input_details  = interpreter.get_input_details()
output_details = interpreter.get_output_details()

print(input_details)
print(output_details)

On TF master with your commit I get same results with 48002 size:
python3 create_tflite_dilated_conv_model.py
Using TensorFlow backend.
WARNING: Logging before flag parsing goes to stderr.
W0506 21:28:29.620581<PHONE_NUMBER>94048 deprecation_wrapper.py:119] From /home/tema/Work/voicesdk/data2/init/antispoof2/wavenent/wn/p3-tf-night/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

W0506 21:28:29.628713<PHONE_NUMBER>94048 deprecation_wrapper.py:119] From /home/tema/Work/voicesdk/data2/init/antispoof2/wavenent/wn/p3-tf-night/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0506 21:28:29.630243<PHONE_NUMBER>94048 deprecation_wrapper.py:119] From /home/tema/Work/voicesdk/data2/init/antispoof2/wavenent/wn/p3-tf-night/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (InputLayer)           (None, 48000, 40)         0
_________________________________________________________________
dilated_conv_2 (Conv1D)      (None, 48000, 40)         3240
=================================================================
Total params: 3,240
Trainable params: 3,240
Non-trainable params: 0
_________________________________________________________________
W0506 21:28:29.653485<PHONE_NUMBER>94048 deprecation_wrapper.py:119] From /home/tema/Work/voicesdk/data2/init/antispoof2/wavenent/wn/p3-tf-night/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

W0506 21:28:29.653728<PHONE_NUMBER>94048 deprecation_wrapper.py:119] From /home/tema/Work/voicesdk/data2/init/antispoof2/wavenent/wn/p3-tf-night/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2019-05-06 21:28:29.657524: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-05-06 21:28:29.657663: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5a7e880 executing computations on platform Host. Devices:
2019-05-06 21:28:29.657677: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-05-06 21:28:29.669086: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1328] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
W0506 21:28:29.717485<PHONE_NUMBER>94048 hdf5_format.py:221] No training configuration found in save file: the model was *not* compiled. Compile it manually.
2019-05-06 21:28:29.719565: I tensorflow/core/grappler/devices.cc:60] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA support)
2019-05-06 21:28:29.719639: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2019-05-06 21:28:29.725718: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:710] Optimization results for grappler item: graph_to_optimize
2019-05-06 21:28:29.725747: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:712]   function_optimizer: Graph size after: 51 nodes (0), 48 edges (0), time = 0.035ms.
2019-05-06 21:28:29.725753: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:712]   function_optimizer: Graph size after: 51 nodes (0), 48 edges (0), time = 0.055ms.
W0506 21:28:29.727740<PHONE_NUMBER>94048 deprecation.py:323] From /home/tema/Work/voicesdk/data2/init/antispoof2/wavenent/wn/p3-tf-night/lib/python3.5/site-packages/tensorflow/lite/python/util.py:237: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
W0506 21:28:29.727875<PHONE_NUMBER>94048 deprecation.py:323] From /home/tema/Work/voicesdk/data2/init/antispoof2/wavenent/wn/p3-tf-night/lib/python3.5/site-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2019-05-06 21:28:29.732639: I tensorflow/core/grappler/devices.cc:60] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA support)
2019-05-06 21:28:29.732706: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2019-05-06 21:28:29.738309: E tensorflow/core/grappler/grappler_item_builder.cc:637] Init node dilated_conv_2/kernel/Assign doesn't exist in graph
INFO: Initialized TensorFlow Lite runtime.
[{'quantization': (0.0, 0), 'name': 'input', 'shape': array([    1, 48000,    40], dtype=int32), 'index': 10, 'dtype': <class 'numpy.float32'>}]
[{'quantization': (0.0, 0), 'name': 'dilated_conv_2/Tanh', 'shape': array([    1, 48002,    40], dtype=int32), 'index': 2, 'dtype': <class 'numpy.float32'>}]

Can you please check this script in your workspace, may be i did something incorrectly?

@gashmish: Thanks a lot for confirming. Even i was also suspecting there might be some more cases missed out. If possible please try to share, the cases where it has not converted. Will try to make solution for converter more generic.
Yes you are right, together 3 PRs #28410, #27867 & #28179 will help the model works fine.
Just to remind you, are you able to reproduce the STRIDED_SLICE problem? If yes would you please share steps or pseudo code to reproduce it.
Thanks a lot for your efforts!

@ANSHUMAN87:
Here is 2 models in archive, almost identical to what we are using. Input is (48000,) and output is (4,).
models.tar.gz
With 3 RPs applied, running model wavenet_classifier.tflite :
ERROR: tensorflow/lite/kernels/strided_slice.cc:152 op_context.input->type != op_context.output->type (2 != 1)
ERROR: Node number 207 (STRIDED_SLICE) failed to prepare.

This is coused by Keras layer Reshape.
In model wavenet_classifier_fixed.tflite Reshape is changed to Squeeze, it runs correctly.
Thank you very much for support!

@gashmish : Sorry i was busy in other tasks. Now i checked your new models shared. But i could not find anywhere BatchToSpace and SpaceToBatch Ops in h5 files which needs to be converted to Dilated Conv. Would you please cross check. Please let me know if i miss anything. Thanks!

@gashmish : Please ignore my last comment i have deleted that.
It was some tool issue.
I am analyzing the issue currently, will update you as soon as possible. Thanks!

Hello. I believe I have a similar issue. My model based on TCN has 3 issues. Last of which is probably similar to what is mentioned here. I would highly appreciate an advice.
First my environment:

I am working with Pycharm, python3, Keras 2.2.4 tensorflow-gpu 1.12.0
I have probelm using convert() method in pycharm, just like other people voicing it in https://stackoverflow.com/questions/52599110/tensorflow-tococonverter-gives-toco-from-protos-error , so I have changed a constant in /home/[MYUSER]/.virtualenvs/[MYPROJECTNAME]lib/python3.6/site-packages/tensorflow/contrib/lite/python/lite_constants.py:  EXPERIMENTAL_USE_TOCO_API_DIRECTLY = True

Now the problem:
I am using keras-tcn library from https://github.com/philipperemy/keras-tcn/blob/master/tcn/tcn.py in ordet to create a Temporal convolutional Neural Network.
My model has 3 issues in conversion to TFLite.

It has problem with Abs() layer, so I have changed code of library in line 25 from:

max_values = K.max(K.abs(x), 2, keepdims=True) + 1e-5
to:
max_values = K.max(K.pow(x*x,0.5), 2, keepdims=True) + 1e-5
It is an ugly solution, but conversion proceeds further.

This code gets error during conversion, whenever nb_stacks is changed from 2 to 1:

new_layer = TCN(kernel_size=3,dilations=[1],return_sequences=False,nb_filters=128,nb_stacks=2)(previous_layer)
Error says : "A merge layer should be called on a list of inputs."

Another error shows up whenever any of the dilation is >1 while kernel_size being >1, so for example:
new_layer=TCN(kernel_size=3,dilations=[2,4],return_sequences=False,nb_filters=128,nb_stacks=2)(previous_layer)
If previous layer had the time dimension equal 31, the error would look like:
Error states: "Check failed: dim_x == dim_y (31 vs. 35)Dimensions must match"

When is the new set of improvements estimated to show up? I will gladly perform an update then. Is there any temporary hack I can apply to go around this issue till then?

@karolbadowski : I will try find a solution for you. By any chance, do you have some small reproducible code or model for your issues? Please share if possible.
And if you think it is related, have you tried with the PRs i shared in this issue links?
Please do let me know the results post the PRs.
Thanks!

@ANSHUMAN87 Hello.  For now I use a hack in which I try to express same shape of TCN neural network with usage of "same" padding instead of "causal". It is not perfect solution, but at least something for a demo that I am preparing.
To make it work I have created own PR to Keras-TCN library: https://github.com/philipperemy/keras-tcn/pull/56
I did not have opportunity to try your PR's yet, as I use prebuild version of TF-Lite from official installation. However I voted here: https://github.com/tensorflow/tensorflow/issues/21526 for your changes to be accepted / reviewed (I see that they are awaiting review from someone with write privilages).
Lightweight example of TCN neural network with "causal" padding can be created with this library: https://github.com/philipperemy/keras-tcn by calling constructor of class TCN().
For example:
input_tensor = Input(shape=(31, 1024))  # it would be input of 31 moments in time, each having 1024 features
temporal_tensor = TCN(kernel_size=3,dilations=[1,2,4,8],return_sequences=False,nb_filters=32)(input_tensor)
output_tensor = Dense( 1 )(temporal_tensor)
m = Model(inputs=[input_tensor], outputs=[output_tensor])
m.compile(optimizer='adam', loss='mse')

@karolbadowski : Thanks for sharing details.
I have verified the TCN conversion & Inference in TFLite.
My 3 PRs #28410, #27867 & #28179 solve the blocking issue you are facing.
So i would suggest, as it is taking long time to get merged, you can setup one build environment of Tensorflow latest code locally, and merge my 3 PRs mentioned above.
Then you can convert the TCN models and will be able to inference as well.
That will be just one time effort.
NOTE: I have used mnist_example TCN to verify my changes, and i believe it coincides with your requirement also.
Please let me know if you need anything else, Thanks!

@ANSHUMAN87 Thanks for the PRs will help with the same issue I'm having too.

Thank you @ANSHUMAN87
Do you know if the change is already merged with release version of TensorFlow (if so, since which one)?

Hello. Do you know for which release version of TensorFlow this change is planned?

Hi @gashmish ,
We are checking to see if you still need help on this issue. We recommend that you upgrade to 2.6 which is latest stable version of TF and let us know if the issue still persists in newer versions.  Thanks!


Deprecate MutableArrayData

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

MutableArrayData provides an API to copy chunks of data from source arrays into an output array, it is extremely flexible, however, its performance leaves a lot to be desired. In particular its reliance on dynamic dispatch per-run not only incurs dispatch overheads, but prevents the compiler from properly vectorising the code. There are also a number of other issues relating to capacity allocation (#347), null mask handling (#1230), dictionary handling (#506)
There has been a fairly concerted effort over the last year or so to move away from use of MutableArrayData:

Improvements to the Array Builders
Ability to construct Array from iterators
Specialized take and filter kernels that only use MutableArrayData as a fallback for nested types

Describe the solution you'd like

I would like to eventually deprecate and remove MutableArrayData. I think the only major blockers to this are:

The filter kernel uses MutableArrayData as a fallback for filtering nested types
The concat kernel uses MutableArrayData
No kernel for interleaving multiple arrays together (#1523)

Describe alternatives you've considered

Additional context

This can be viewed through the lens of porting @jorgecarleitao 's ideas from arrow2 and bringing them into arrow-rs

Hm, except for MutableArrayData, is there a way to easily copy arrays?

The hope was that all use-cases would be covered by a combination of the take, filter and interleave kernels

The most close one is take. But I quickly take a look and seems it is not as efficient as MutableArrayData for simple copy usage, although I don't run a formal benchmark to verify it.

I think I have made peace with the existence of MutableArrayData


App context refactor

This PR refactors app context to use MutableSharedFlow. This allows classes that depend on Context to wait until the application Context has been captured by ApplicationContextInitializer. This PR also sets the default Cache implementation to SharedPreferencesCache, which is now possible without the users having to explicitly provide Context.

After further discussion with wider Android eng group, we decided that it's best to let users provide Context rather than having auto-initialization. There are caveats with auto-initialization which can pose issues:

User-defined app initializers can run before our own initializer. If users end up using our APIs before our SDK is initialized, inside their initializer, then that will result in the app crashing.
Using a Flow<Context> variable instead of Context for storing Context avoids the crash. But, it can still result in the app hanging if the user calls runBlocking { // SDK APIs } during their initializer. This bug shouldn't happen if the users follow setup properly, but it's a difficult issue to debug, should users run into this. The app never crashes or sends an ANR in this case.

Because of these reasons, it's a better idea to continue having the users explicitly provide Context during SDK initialization.


[Feature] data-table to allow observable object

Feature Request
I'm using AngularFire2 to pull in my data like so:
people : FirebaseListObservable<any[]>;
constructor(private af: AngularFire){
this.persons = af.database.list('people');
}

When using an *ngFor with the async pipe, this is really easy to implement.
<md-list-item *ngFor="let person of persons | async">
However, I can't seem to get this to work with td-data-table. I'm not sure if this is a feature request or if there is some other way to accomplish observable data in a data-table. If so, I'd be happy to submit a PR to update the documentation and code examples.
What is the motivation / use case for changing the behavior?
The ability to populate a data-table with observable data so the table data is dynamic and reactive (aka plays nice with Firebase).
Which version of Angular and Material, and which browser and OS does this issue affect?
"angular-cli": "1.0.0-beta.19-3",
"@covalent/data-table": "0.9.1",

Should be the same:

   ... you columns ...

Yep, as @mxii said, you need to use the atomic parts of data-table to do so.

Yes, I was able to get that working by using the atomic parts of the data-table. That's great!
I was hoping to take advantage of properties

[sortable]="true"
[sortBy]="sortBy"
[sortOrder]="sortOrder"
(sortChange)="sort($event)"

It looks like it's possible to achieve these features by implementing them manually, I just need to learn how. Thanks!

What is the reason it's not possible to convert the observable to an array for the data: any[] property  to use?
people : FirebaseListObservable<any[]>;
public testArray: Array<any[]>;
constructor(private af: AngularFire,) {
      this.people.subscribe(
        data => {
         let mappedData: any = data.map(this.mapFunc);
          this.testArray = mappedData;
        });
}

This appears to make testArray an array just as the documentation uses but whenever I use

it fails with a "Unexpected token u in JSON" console error.

well, from what i see [data] should be testArray in that example.. not sure if it was a misstype.
But yeah, if the testArray variable is an array of object, it should be rendered correctly.

I was able to get this working with the following:
people: FirebaseListObservable<any[]>;
peopleList: any[] = [];

    this.people = af.database.list('/people');
    this.people.subscribe(peopleDatObj => {
      this.peopleList = peopleDatObj;
    });

This issue causing the JSON error was solved by adding peopleList: any[] = [];. I assume the JSON parsing didn't like having an empty undefined array and that was firing before my observable returned data.
There are still issues with observables though. Paging also fires before there is data so it shows "1-0 of 0" results even though there are rows in the table. Any thoughts on how to resolve that?

huh, we might need to fix that for an empty array either way. For paging you need to set the total manually in the [total] input, since paging and search components are agnostic of the data-table component

I got this working and would love to contribute to the documentation. One last hiccup I'm trying to resolve. After adding paging and filter() method, I don't get any data on initial load, however after I change the number of rows to display in the pager, my data loads into the table perfectly.
Any idea how to initiate the pager and sorting after data is loaded?
    <md-card tdMediaToggle="gt-xs" [mediaClasses]="['push']">

        Data Table using Data Property

      >

    </md-card>

export class UsersComponent {

  columns: ITdDataTableColumn[] = [
      { name: "favoriteFruit",  label: 'Name' },
      { name: 'email', label: 'Email' },
      { name: 'registered', label: 'Start Date', numeric: true, },
    ];

  people: FirebaseListObservable<any[]>;
  peopleList: any[] = [];

  filteredData: any[] = this.peopleList;
  filteredTotal: number = this.peopleList.length;
  searchTerm: string = '';
  fromRow: number = 1;
  currentPage: number = 1;
  pageSize: number = 5;
  sortBy: string = 'email';
  sortOrder: TdDataTableSortingOrder = TdDataTableSortingOrder.Descending;

  constructor(private af: AngularFire,
              private _titleService: Title,
              private _router: Router,
              private _dataTableService: TdDataTableService) {

    this.people = af.database.list('/');
    this.people.subscribe(peopleDatObj => {
      this.peopleList = peopleDatObj;
      });
}

  ngOnInit(): void {
    this._titleService.setTitle( 'Intensive Users' );
    this.filter();
  }

  sort(sortEvent: ITdDataTableSortChangeEvent): void {
    this.sortBy = sortEvent.name;
    this.sortOrder = sortEvent.order;
    this.filter();
  }
  search(searchTerm: string): void {
    this.searchTerm = searchTerm;
    this.filter();
  }
  page(pagingEvent: IPageChangeEvent): void {
    this.fromRow = pagingEvent.fromRow;
    this.currentPage = pagingEvent.page;
    this.pageSize = pagingEvent.pageSize;
    this.filter();
  }
  filter(): void {
    let newData: any[] = this.peopleList;
    newData = this._dataTableService.filterData(newData, this.searchTerm, true);
    this.filteredTotal = newData.length;
    newData = this._dataTableService.sortData(newData, this.sortBy, this.sortOrder);
    newData = this._dataTableService.pageData(newData, this.fromRow, this.currentPage * this.pageSize);
    this.filteredData = newData;
  }
}

Awesome that you got it working! :+1:
So, the problem with the initial load is that you are doing it in the constructor and filtering in ngOnInit which dont guarantee that they will be executed in that order.
So you need to do remove the loading part from the constructor (since its frowned upon in angular either way) and do it all in the ngOnInit lifehook component cycle.
Example:
constructor(private af: AngularFire,
              private _titleService: Title,
              private _router: Router,
              private _dataTableService: TdDataTableService) {
}

  ngOnInit(): void {
    this._titleService.setTitle( 'Intensive Users' );
    this.people = af.database.list('/');
    this.people.subscribe(peopleDatObj => {
      this.peopleList = peopleDatObj;
      this.filter();
    });
  }

This way you are following the angular standard and style guide and executing filter() once the callback returns with the necessary data.

That worked great, thanks! I'm still learning JS so sorry for the newbie questions.
Is the empty array need for the observable a bug?
Think this is worthy of an update to the documentation? If so, thoughts on showing the demo with an observable string like I am from Firebase?
If not, you can mark this issue as closed. Thanks again!

I will keep this open since that is a legit bug. The data table shouldnt break even if the [data] input is undefined nor empty.
I will add a fix for this scenario in 0.10.

Already added a fix for the undefined array here https://github.com/Teradata/covalent/pull/188/commits/e760d16a1f0885ada8f37177385f5876136fc181

Closing this since the fix for an undefined array is already in review and will be part of 0.10.

I wonder why not to return reactive data from the sort and filter options? I think even the paginate could have benefits from using the observable pattern.

Yeah, that would require some discussion. Please open a new issue regarding that specifically 😄


Updated Adobe DNG Converter version to 9.1.

Thank you for the contribution. It was merged directly as fdbfec81c429d7ff8df2873118942d611914ff6e to keep commit history cleaner. Your contribution is still credited to you.


Weird behaviour when there is a JSON object conaining a "data" attribute

Hi,
still love your work and use it in quite alot of projects! :)
But I guess there is something wrong when I try to GET an "apprequest", I only get the contents of the "data" attribute which is "tracking data here" but nothing else. I guess its some kind of conflict between "data" variables in your facepy Code and "data" attributes inside the json structure.
{
"id": "222162114516955",
"application": {
"name": "-----------------------------------",
"canvas_name": "-----------------",
"namespace": "-------------------",
"id": "--------------------"
},
"to": {
"name": "------------",
"id": "------------"
},
"from": {
"name": "----------",
"id": "----------"
},
"data": "tracking data here",
"message": "---------------------------",
"created_time": "2011-11-03T11:10:35+0000",
"type": "apprequest"
}
I try to figure out whats wrong but Im not the best coder. ;)

Ah, I guess its that one:
try:
obj = result['data']
except (KeyError, TypeError):
obj = result

Hi @Morpho,
I'm glad you find facepy useful! As you have already discovered, the library discards properties other than data because it's typically the only attribute you'd be interested in. I haven't had the opportunity to use app requests yet and so it's hard for me to determine whether that applies in this case, too; would you mind elaborating on whether and why the remaining properties are of interest to you?

Hi Johannes,
yes, the other fields besides of "data" supplied in an apprequest are absolutely necesary, because they tell you for example who send the request and to whom the request was send. And requests are all about sending them to a specific! amount of people, so you definetly need the "from" and "to" attributes. (compare http://developers.facebook.com/docs/reference/dialogs/requests/)
In my copy of facepy I removed that part of discarding everything else when there is a data attribute present. I think thats better.
Its your decision to choose whether this is good or not for everybody else, but in my case I really need it.

Thanks for clearing that up, @Morpho, and please feel free to open a pull request. I'll give this some thought, but in the end I'll probably pull it.

For the fun of it (and because I think it makes sense) I created a fix for this.
It breaks the api and should probably be released as part of a 0.5 release.

Thanks so much @madssj ! Its a bit embarrassing but I dont even know how to create a pull request.

You're welcome @Morpho - github has an extensive help page on the subject :)

Thanks guys. I'm going to merge this and bump the version to 0.5.


Generate keywords

Hey @leodr, great work on this package.
It would be awesome to populate the data with keywords that would help to better search for emojis.
I found emoji-data that does exactly this using CLDR dataset.
Would you be OK to migrate to this dataset in order to have these precious keywords? I can work on a PR.

To explain a bit more, your package is used on the Emoji Raycast extension and it lacks some predictability sometimes.
Example:

I would expect to have at least one "shocked" emoji, right?

Here is a sample of the data
Search for "shocked"

To explain a bit more, your package is used on the Emoji Raycast extension and it lacks some predictability sometimes.
Example:

I would expect to have at least one "shocked" emoji, right?

That's cool, love seeing it in use there!
I'd be open to expanding the data with keywords, but I wouldn't like to deviate from the official source, maybe we could merge in the keywords from the list you provided just like we currently do with the GitHub shortcuts?

I'd be open to expanding the data with keywords, but I wouldn't like to deviate from the official source, maybe we could merge in the keywords from the list you provided just like we currently do with the GitHub shortcuts?

Yeah, that's definitely the way to do this. Are you interested in digging in or should I give it a spin?

I'd be open to expanding the data with keywords, but I wouldn't like to deviate from the official source, maybe we could merge in the keywords from the list you provided just like we currently do with the GitHub shortcuts?

Yeah, that's definitely the way to do this. Are you interested in digging in or should I give it a spin?

Im on holiday right now, but I could review your PR and release a new version if you want to implement this feature.


ACR and AKS integration gives resource does not support http method 'PUT'

Following on #144 I updated the role assignment to perform a nil check on aks.Identity.PrincipalID. However, now the issue seems to be the HTTP method used to perform the role assignment is not supported.
Relevant code:
_, err = authorization.NewRoleAssignment(ctx, "aks_acr", &authorization.RoleAssignmentArgs{
    Scope:              acr.ID(),
    RoleAssignmentName: pulumi.Sprintf("%v/Microsoft.Authorization/%v", "pfmbohdev", id.Result),
    RoleDefinitionId:   pulumi.Sprintf("/subscriptions/sub/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d", sub),
    PrincipalId: aks.Identity.PrincipalId().ApplyString(func(s *string) string {
	if s == nil {
	    return "0"
	}

	return *s
    }),
})

And error returned:
azure-nextgen:authorization/v20180901preview:RoleAssignment (aks_acr):
    error: autorest/azure: Service returned an error. Status=405 Code="" Message="The requested resource does not support http method 'PUT'."

AFAIK RoleAssignmentName should be simply a GUID, try removing Sprintf there

Yes, you're right. Changing RoleAssignmentName to a regular GUID worked. I modeled this after arm2pulumi so I think there might be an issue with that.


Introduce type checking

What
I need to introduce a type checking system to the current library.
Why
Strong types is a must.
How

[ ] Install TypeScript dependencies
[ ] Add TypeScript compilation to the build pipeline
[ ] Test

The size of this library is rather small to benefit from type checking. That feels as an overkill in the current conditions, therefore, I am closing this task.


[XNIO-320] test case

Test case for:
7.1.z JIRA https://issues.jboss.org/browse/JBEAP-14286
XNIO JIRA https://issues.jboss.org/browse/XNIO-320
llegalArgumentException: Parameter 'address' may not be null if invalid-hostname does not resolve to an IP, then you will see the error.
Correct behavior is UnknownHostException

Could anyone please review the testcase?

@dmlloyd what do you think about the location of this test? Personally I think it should just go into the XNIO test suite, it seems like overkill to have a WFLY test for it.

I agree, it should go into the XNIO test suite.

Closing, as I think this is the wrong location for this test


Potential CivModCore race condition fixes

Addresses two possible race condition scenarios:
A first race condition with unloadChunkCoord in the following scenario:

Thread A checks coord.isUnloaded() inside the synchronized block towards the end of the method and finds that it is unloaded.
Thread B loads the chunk before Thread A removes the chunk from the metas map.
Thread A removes the chunk from the metas map, even through it's now loaded.

Attempted to be fixed by preventing the chunk from being loaded while we're checking it and potentially removing it from metas.
The second race condition is to do with loadChunk, where:

Thread A calls loadChunk
Thread B unloads the chunk before Thread A update's the chunk's status
Thread A updates the chunk's status to loaded, even though it's now unloaded.

Attempted to be fixed by checking inside the loadChunk method to ensure the chunk has not been unloaded after it was loaded. We're holding a lock on the metas map to prevent other threads from unloading the chunk during the check.

I don't think either of your race conditions will happen because getChunkCoordalso has a synchronised (metas) before interacting with the metas map.


Remove/unify variables in Constants.java/config.json

Making notes on some that would be nice to remove:
INTENSITY_THRESHOLD?
LINE_SEARCH_WIDTH

Would also be nice to have a nested structure so that people can do multiple fiducials, but that will be a lot of work for later (don't want to make it an issue right now)

So now the Constants.java file should read the values from config.json?


Performance issue with shadowRadius

New Issue Checklist

I have read the guidelines for contributing and I understand:

[x] My issue is happening in the latest version of SideMenu.
[x] My issue was not solved in the README.
[x] My issue can not be answered on stackoverflow.com.
[x] My issue is not a request for new functionality that I am unwilling to build and contribute with a pull request.
[x] My issue is reproducible in the demo project.

Issue Description

My project has a lot of ui components and layer.shadowRadius with its live rendering is causing some performance issue when open and close the menu, maybe switch to layer.shadowPath could help?

Hi @xuaninbox,
This is the first reporting of this performance issue, which may suggest it's not a problem for the vast majority of developers. Does performance improve when the shadow is turned off? Can you test switching to a layer.shadowPath to see if that improves performance?

@xuaninbox I'm closing this for now. Please add any additional insights you may have and we can reopen this as necessary. Thanks!


How to make carouselview clickable?

I'm going to move to next page, when carouselview is clicked.
How can I implement this feature?

Here is the solution.
CarouselViewPager pager = (CarouselViewPager) view.findViewById(R.id.containerViewPager);
final GestureDetector tapGestureDetector = new GestureDetector(context, new TapGestureListener());
pager.setOnTouchListener(new View.OnTouchListener() {
public boolean onTouch(View v, MotionEvent event) {
tapGestureDetector.onTouchEvent(event);
return false;
}
});
class TapGestureListener extends GestureDetector.SimpleOnGestureListener{
@Override
public boolean onSingleTapConfirmed(MotionEvent e) {
// Here you can do anything you want.
return false;
}
}

I've dropped a PR with this feature, hope the author reviews and accept it !


Allow visitors to submit email directly to hubspot instead of google …

…form
there's an option to send a follow-up/confirmation email after sign up but i haven't configured that: https://app.hubspot.com/forms/46270089/editor/bb54ab39-951b-423d-b141-9182e623c8fa/edit/automation

updating the font color is a paid feature -- not sure if it's worth changing the dark background color or leave it as is for now


something-went-wrong-page craeted

Summary
A server error page is created to which the user will be redirected if there is an error in the RestaurantView, RestaurantList, CarouselComponent or RestaurantFilterComponent requests

the REACT_APP_API_URL environment variable was assigned to the URL variable and consumed from there


Exporting APIs

There is a branch that seems to work with exporting APIs. Can this be pushed to a maven repository? I've tested it and it seems to work well enough

Thanks for bringing this up. Yes that is almost working but needs a few tweaks for it to be a bit more production-ready. We'll prioritize this as several people have asked for it.

Is this repository still active?

@stewrutledge, it's being revived after some quiet period. You'll be seeing the functionality in the coming couple of weeks.

@stewrutledge this is now resolved. With version 0.1.4 you should be able to export transactions. Thanks!


stage0: minor wording fixes

Minor rewording after CRI merge, cleaning up "appz" occurrences.

/cc @squeed

Rephrased a little bit with @euank and @jonboulle wording.

lgtm

Semaphore failure is the known kvm stop flake.


Fix an incorrect auto-correct for Style/HashConversion

This PR fixes the following incorrect auto-correct for Style/HashConversion when Hash[] as a method argument without parentheses.
% cat example.rb
do_something Hash[a: b, c: d], 42

% bundle exec rubocop --only Style/HashConversion -a
(snip)

Inspecting 1 file
C

Offenses:

example.rb:1:14: C: [Corrected] Style/HashConversion: Prefer literal
hash to Hash[key: value, ...].
do_something Hash[a: b, c: d], 42
             ^^^^^^^^^^^^^^^^

1 file inspected, 1 offense detected, 1 offense corrected

Before
Auto-corrected to invalid syntax.
% cat example.rb
do_something {a: b, c: d}, 42

% ruby -c example.rb
example.rb:1: syntax error, unexpected ':', expecting '}'
do_something {a: b, c: d}, 42

After
Auto-corrected to valid syntax.
% cat example.rb
do_something({a: b, c: d}, 42)

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[ ] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Thanks!


[BUG] "ERR_STRING_TOO_LONG" during export

Context

ElasticDump version: 6.72.0
Elasticsearch version: 7.13.3
Node.js version: (I'm running elasticdump from docker image, using the version of Node.js provided)
The following is the full command being used, and the resulting error:

$ docker run --rm -ti --network host -v /tmp:/tmp elasticdump/elasticsearch-dump \
   --input=http://localhost:9200/net-log-2021 \
   --output=/tmp/net-log-2021.ndjson \
   --type=data

Fri, 03 Sep 2021 22:01:57 GMT | starting dump
buffer.js:798
    return this.utf8Slice(start, end);
                ^

Error: Cannot create a string longer than 0x1fffffe8 characters
    at Buffer.toString (buffer.js:798:17)
    at Request.<anonymous> (/usr/local/lib/node_modules/elasticdump/node_modules/request/request.js:1128:39)
    at Request.emit (events.js:375:28)
    at Request.emit (domain.js:470:12)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/elasticdump/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (events.js:481:28)
    at IncomingMessage.emit (events.js:387:35)
    at IncomingMessage.emit (domain.js:470:12)
    at endReadableNT (internal/streams/readable.js:1317:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  code: 'ERR_STRING_TOO_LONG'
}

Describe the bug
While trying to use elasticdump to export the data of an index using the above command, it runs for a bit and then returns the above error.
To Reproduce
I'm not able to provide the dataset from the index since I can't export it, but this index does have some fields where the value may be rather long.
Other comments
Searching for this particular error online highlights that this is likely a Node.js error related to a fixed limitation on the size of strings in Node.js.
If this is a limitation in Node.js, is there a way I can get around this to successfully export the data?

You're dumping too much data. Try using the split function

Related to https://github.com/nodejs/node/commit/d2b9e7cb01166da0e6cfd77d1cf362b8d49c6c68

I also ran into this issue. After a lot of experimentation I was able to complete a dump using --limit=1000 --fileSize=100mb. No idea if this works for others.

It's works because you're segmenting the document into smaller chunks and not reading the large string into memory.


Publish to Docker Hub failing with "libcrypt.so.1: cannot open shared object file"

This means the Datasette 0.59.1 release has not been published to Docker Hub.
Here's where that failed: https://github.com/simonw/datasette/runs/3991043374?check_suite_focus=true
Preparing to unpack .../libc6_2.32-4_amd64.deb ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Checking for services that may need to be restarted...
Checking init scripts...
Unpacking libc6:amd64 (2.32-4) over (2.28-10) ...
Setting up libc6:amd64 (2.32-4) ...
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
dpkg: error processing package libc6:amd64 (--configure):
 installed libc6:amd64 package post-installation script subprocess returned error exit status 127
Errors were encountered while processing:
 libc6:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update &&     apt-get -y --no-install-recommends install software-properties-common &&     add-apt-repository "deb http://httpredir.debian.org/debian sid main" &&     apt-get update &&     apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite &&     apt-get remove -y software-properties-common &&     apt clean &&     rm -rf /var/lib/apt &&     rm -rf /var/lib/dpkg/info/*' returned a non-zero code: 100

Same problem when I attempted to publish using the "Push specific Docker tag" workflow: https://github.com/simonw/datasette/runs/3991059912?check_suite_focus=true

You may have just been talking to yourself here, but I found your documentation of the process incredibly useful when I hit the same error myself. Thanks! 🌈


Allow for 'main' as branch name value

Part of https://github.com/pulumi/home/issues/933
This PR should allow us to generate a Workflow for building on a default branch named main.
I changed the class name as well to better illustrate the purpose of addressing a repo's default branch.

Tried this out on Tailscale and it generates both a main.yml as well as a master.yaml.
Instead of hard coding "master" I named the job by its inherited job name, since they were the same anyway.
Either way, the danger of doing this is that when migrating from master to main, folks may accidentally trigger both runs as they would exist in parallel. I would love to add actual logic so that we don't have equally valid runs on two "default" branches ever.

Ok I guess this puts us in better shape going forward since we actually have a workflow file we can delete later :)


add parrallel runner for sdc tests

Hello @KsanaKozlova! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file sdc/runtests.py:

Line 9:1: W293 blank line contains whitespace

In the file sdc/testing/__init__.py:

Line 23:89: E231 missing whitespace after ','
Line 38:1: E303 too many blank lines (4)

In the file sdc/tests/__init__.py:

Line 5:1: E302 expected 2 blank lines, found 1


New version of GEOS-Chem FullChem

Codecov Report

Merging #33 (cb62de9) into main (16db2d5) will decrease coverage by 9.92%.
The diff coverage is 0.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
- Coverage   66.55%   56.63%   -9.92%
==========================================
  Files           9       10       +1
  Lines         885     1040     +155
==========================================
  Hits          589      589
- Misses        296      451     +155

Files Changed
Coverage Δ

src/geoschem_ratelaws.jl
0.00% <0.00%> (ø)


Cargo dependency cannot recognize ~ as home path in dependency's path field

Problem
I have a dependency on my local file system:
❯ cat Cargo.toml
[package]
name = "cargo-path-bug"
version = "0.1.0"
edition = "2021"

[dependencies]
cargo-path-bug-lib = { path = "~/Temp/cargo-path-bug-lib/", package = "cargo-path-bug-lib"}

But when I run cargo run, it recognize ~ as a dir, not my home path.
❯ cargo run
error: failed to get `cargo-path-bug-lib` as a dependency of package `cargo-path-bug v0.1.0 (/home/exec/Temp/cargo-path-bug)`

Caused by:
  failed to load source for dependency `cargo-path-bug-lib`

Caused by:
  Unable to update /home/exec/Temp/cargo-path-bug/~/Temp/cargo-path-bug-lib

Caused by:
  failed to read `/home/exec/Temp/cargo-path-bug/~/Temp/cargo-path-bug-lib/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

Steps
No response
Possible Solution(s)
request to recognize ~ as home path.
Notes
No response
Version

Could you help me understand why you want to use user-relative paths?  In my mind, the use case is limited because no one else can use that.    For local prototyping or patching, I would assume relative paths would work just as well.  At worse, you can fully specify the path.
Support for ~ is not built-in to the OS but is something apps have to manually support.  Technically, adding support would be a breaking change as ~ is allowed in paths.

Could you help me understand why you want to use user-relative paths?

emmm.I just got used to doing it this way.

I think windows systems have ~ too, it's %USERPROFILE%, may like C:\Users\John

Yes, most software that supports ~ maps it to the user profile. I was more so highlighting that users don't normally put files there and so

Its less convenient as more nesting is needed
Cloning a repo that uses ~ will require unusual filesystem layout for a windows user


counter_culture_fix_counts on has_many association results in error

I have a counter on a has_many relationship and it works perfectly fine. However, when I run the CollectionsContent.counter_culture_fix_counts command I get the following error:
Content Load (1.0ms)  SELECT  "contents".id, "contents".id, COUNT("collections_contents".)*1 AS count, "contents".contents_count FROM "contents" LEFT JOIN "collections_contents" AS collections_contents ON "contents".id = collections_contents.collection_id GROUP BY "contents"."id" ORDER BY "contents"."id" ASC LIMIT $1  [["LIMIT", 1000]]
Traceback (most recent call last):
        1: from (irb):1
ActiveRecord::StatementInvalid (PG::SyntaxError: ERROR:  syntax error at or near ")")
LINE 1: ...".id, "contents".id, COUNT("collections_contents".)*1 AS cou...
                                                             ^
: SELECT  "contents".id, "contents".id, COUNT("collections_contents".)*1 AS count, "contents".contents_count FROM "contents" LEFT JOIN "collections_contents" AS collections_contents ON "contents".id = collections_contents.collection_id GROUP BY "contents"."id" ORDER BY "contents"."id" ASC LIMIT $1

Classes
class Collection < ApplicationRecord
  has_many :collections_contents
  has_many :contents, through: :collections_contents, dependent: :destroy
end

class Content < ApplicationRecord
  has_many :collections_contents
  has_many :collections, through: :collections_contents, dependent: :destroy
end

class CollectionsContent < ActiveRecord::Base
  belongs_to :content
  belongs_to :collection
  counter_culture :collection, column_name: :contents_count
end

Is there something wrong with the code?

Perhaps that model doesn't have a primary key set? That's what the reconciler uses.
I'm happy to help dig on this more, but you'd have to open a pull request with a failing test case for me to be able to reproduce this.

Thanks for the response,  @magnusvk. You are absolutely correct, the model didn't have a primary key which caused the error. Thanks for all of your hard work

Only want to add the corresponding migration for that issue, to save time for others:
add_column :collections_contents, :id, :primary_key

BTW: Thank you for this gem 👍


Github workflows: re-implement a no-shared build

We do this both on Ubuntu and MacOS X

And now you have to solve the duplicate symbols problem.

Re-created in #14755

For testing purposes, I've a merge of this and #14755 in https://github.com/levitte/openssl/tree/fix-11281%2B%2Bworkflow-reimplement-no-shared

Rebased just to prove that it now builds without failure

This pull request is ready to merge

Merged
4a95b70d1e3df791ea569e94067cf0bec1f69557 Github workflows: re-implement a no-shared build


WP6.7 - GAW

Added as a discussion item for the upcoming ET-ACDM 2023-02-15 meeting.

Discussion in https://github.com/wmo-im/et-acdm/issues/20.  Plan is to have a draft by our ET-ACDM next meeting.

ET-ACDM update in https://github.com/wmo-im/et-acdm/issues/20#issuecomment-1582723300
Current target is to have a draft for WTH inclusion by end July 2023.


README for T1/T2 participants

Looks good, thanks! Is it intended that this PR also includes some plots and result files?

Yes, wanted to move T1/T2 results to this folder.


Screenshot

Hi @NHP95 ,
I am able successfully connect to the reportportal. Thank you.
is it possible to send the screenshot as well? Please help
Thanks

You can attach screenshots by directly calling captureScreenShotForReportportal keyword inside your test cases.

yeah @NHP95 , I am about update that i figured it out.
Thanks,
Siva


geocode.xyz - out of credits

All my recent activities were showing as "Unknown Area" so I logged into my geocode.xyz account and found that I am out of credits -

Not sure why this is happening as I only use Strava a couple of time a month.

Hi @BertrumUK
This is due to the activites refreshing the data everytime there is an update or your HA instance is restarted.
I've added some logic in the latest releases 3.2.27+ that will use the location from the activity segment first and fallback to the geocode.xyz service if its not found.

Thanks Tim. Working great so far.


"Get the specification" link on openthread.io is dead (404)

To update link to: https://www.threadgroup.org/support#specifications
@jrhodie FYI - I saw you did some work on documentation recently. Maybe you know who could fix this?

@jrhodie Can you fix this?  It's the one at the top of the home page of openthread.io.
Maybe also check if the same link exists elsewhere on the site.

@Vyrastas @EskoDijk Sorry I missed this earlier...fixing it now.

ok, should be working now with the new link provided by @EskoDijk    thanks!


Requirejs module name: make it empty so that we can use a different one?

Hi,
I'm just integrating moment.js in my requirejs project. I'm having an issue because I want to include momentjs independently from the general config.js (so I don't want the rest of the application to know that I'm including this lib).
So, here is my code for my project:
define(function(require, exports, module){
    'use strict';

    var moment = require('./moment.min');
    // ...

This does not work, because moment.js define a requirejs module, named "moment", and not "./moment.min" (or whatever I want to call it). Although a "path" config makes it work, it's something that I want to avoid.
Just changing ( here: https://github.com/moment/moment/blob/develop/moment.js#L1777 ):
    if (typeof define === "function" && define.amd) {
        define("moment", [], function () {
            return moment;
        });
    }

by:
    if (typeof define === "function" && define.amd) {
        define([], function () {
            return moment;
        });
    }

makes it work.
My question is: is there a particular reason why a name is explicitely defined? Does it break anything?
Thanks!
Kev

@radekn  @cowwoc etc
https://github.com/moment/moment/pull/1963

@wyantb Hurray! Thank you!


218331851 magazi

needs work on naming convention and the overall structure. Please fix this for the next PR, Approved!


fix(auction): fixes time display (again)

The type of this PR is: Bugfix
Re: https://github.com/artsy/force/pull/9648 — somehow the change got removed.

Thank you!


About learning rate decay

Hello,
Is the GAN trained with a fixed learning rate?
The discriminator LR: 0.0004
The generator LR: 0.0001
Are these learning rates decayed? If so, where may I find the implementation?

???


[Question] How does this work?

Does this enables spices to be written in TypeScript while being read by CJS as normal Javascript?

Are this a framework for developing spices or an extension for Cinnamon itself?

For the first question, yes if you transpile to JS.
For the second question, this just enables development of spices using Typescript. More specifically it provides typescript declarations for libraries that are provided out of the box for spices in CJS under imports.*.

So, spices' devs could write the "frontend" in vanilla CJS while imports.*'ing most of the code (the "backend" - quotes with an extra grain of salt pls) in TS + types-cjs declarations?

You don't need this library for vanilla (C)JS. You can also write an applet/extension in TS without it with a dummy declaration like this:
declare global {
    const imports: any;
}

but obviously it has some drawbacks.
So this library is only for providing declarations for that namespace for correct type-checking for Typescript.


Bug report

1:(obj_piranhaplant_side_right)and(obj_piranhaplant_side_left)cant hurt mario:

2:raccoon mario cant use tail to open spike qblock and winged qblock

Fixed in


RuntimeError: grad can be implicitly created only for scalar outputs

`import warnings
warnings.filterwarnings('ignore')
from torchvision import models
import numpy as np
import torch
import cv2
import yaml
import requests
from models.experimental import attempt_load
from pytorch_grad_cam import GradCAM, EigenGradCAM
from pytorch_grad_cam.utils.model_targets import ClassifierOutputTarget
from pytorch_grad_cam.utils.image import show_cam_on_image,
deprocess_image,
preprocess_image
from PIL import Image
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
weigths = torch.load('yolov7.pt')
model = weigths['model']
model = model.half().to(device)
_ = model.eval()
#image = cv2.imread(r"/home/harish/yolov7/figure/25.jpg")
image_url = r"/home/harish/yolov7/figure/25.jpg"
img = np.array(Image.open(image_url))
img = cv2.resize(img, (224, 224))
img = np.float32(img) / 255
input_tensor = preprocess_image(img, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
input_tensor = input_tensor.half().to(device)
#The target for the CAM is the Bear category.
#As usual for classication, the target is the logit output.
#before softmax, for that category.
targets = [ClassifierOutputTarget(1)]
target_layers = [model.model]
with GradCAM(model=model, target_layers=target_layers) as cam:
grayscale_cams = cam(input_tensor=input_tensor, targets=targets)
cam_image = show_cam_on_image(img, grayscale_cams[0, :], use_rgb=True)
cam = np.uint8(255grayscale_cams[0, :])
cam = cv2.merge([cam, cam, cam])
images = np.hstack((np.uint8(255img), cam , cam_image))
cv2.imwrite("/home/harish/yolov7/figure/heatmap.jpg", images)`
ERROR Message:
Traceback (most recent call last):
File "visulaization.py", line 39, in
grayscale_cams = cam(input_tensor=input_tensor, targets=targets)
File "/home/harish/.local/lib/python3.8/site-packages/pytorch_grad_cam/base_cam.py", line 188, in call
return self.forward(input_tensor,
File "/home/harish/.local/lib/python3.8/site-packages/pytorch_grad_cam/base_cam.py", line 84, in forward
loss.backward(retain_graph=True)
File "/home/harish/.local/lib/python3.8/site-packages/torch/tensor.py", line 488, in backward
torch.autograd.backward(
File "/home/harish/.local/lib/python3.8/site-packages/torch/autograd/init.py", line 190, in backward
grad_tensors = make_grads(tensors, grad_tensors, is_grads_batched=False)
File "/home/harish/.local/lib/python3.8/site-packages/torch/autograd/init.py", line 85, in _make_grads
raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs
Error is with loss.backward i think. help is highly appreciated.

We can't use the Classifier Output Target, since it's an object detection network with multiple outputs.
I'm also not sure if Yolo7 gives you a way to compute the gradients.
The documentation has examples for other YOLO version, I would start from there and then try to adapt to YOLO7.

Have the same problem but still don‘t know how to solve it yet.


feat: Added missing annotations for zeebe operate service

What?
I have added the ability for operate service to have its own custom annotations.
Why?
Having ONLY common annotations is not enough. For example if we want to expose each service with a custom domain name

Thanks @michelefa1988 for your contribution I will take a look at it next week, currently in sick leave.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@Zelldon  I resigned, I think it should be ok now,...... wierd


How to send and receive messages at the same time on server

Hi, I want to implement a function which can  send and receive messages to a client in one connection at  the same time on server. However, in the the ' while True' loop of the recv logic, the 'websocket.recv()' in fact blocks the loop.
async def echo(websocket, path):
    while True:
        recv_text=await websocket.recv()
      #I don't know how to add the send logic here
      #bcause I don't know how to how to make the websocket.recv() none-blocking

asyncio.get_event_loop().run_until_complete(
    websockets.serve(echo, 'localhost', 8765))
asyncio.get_event_loop().run_forever()

I tried some ways to solve the question, using multiple tasks, threadings but all failed.
Finally it comes out as follows
async def echo(websocket, path):
    while True:
        try:
            recv_text=await asyncio.wait_for(websocket.recv(), timeout=0)
        except (asyncio.TimeoutError, ConnectionRefusedError):
            print('timeout')
        websocket.send('some other messages')

asyncio.get_event_loop().run_until_complete(
    websockets.serve(echo, 'localhost', 8765))
asyncio.get_event_loop().run_forever()

It works. However,  I'm confused why it works.  Is there a more elegant way to realize the function? Hope I made myself clear.

The "getting started" documentation covers this use case: https://websockets.readthedocs.io/en/stable/intro.html#both

@aaugustin That link is invalid now, does documentation for this still exist somewhere?

https://websockets.readthedocs.io/en/stable/howto/patterns.html#consumer-and-producer


Credentials have no easy way to access its attributes

Currently when you have a credential record it is not easy to access its attributes. You need to either:

parse the preview from the offer message
parse the raw values from the credential message
retrieve the credential from indy storage

Aries Framework .NET just stores the attributes in the record. I think we should do the same

I'm still curious if it's possible to utilize credentialPreview for that

I think that's doable. We can add a connivence method to the credential record that returns the credential preview (or something else based on the credential preview)

My concern is security. I'm not sure if the user can rely on data in credentialPreview, because it could differ from data in actual credentialOffer/credenital.

Well it should be trustable. If the issuer makes an offer, the holder requests it and then issues a different credential it should throw an error and not acknowledge it.
If we don't reject that at the moment, we should add a check for it.

Hmm. I assumed that libindy takes care of consistency between credOffer, credReq and cred, therefore we wouldn't need to validate if the received credential matches the offer. Then we just would need to validate credentialPreview against credentialOffer and stores the credentialPreview as attribute in credentialRecord

How to format credentials:
{
	// metadata contains credentials format specific stuff
	"metadata": {
		"credentialDefinitionId": "creddefid"
	}
	"claims": {
		"name": "name"
	}
}

I'll add a few utility methods such as returning the attributes in a properly cased or returning them alphabetically
@JamesKEbert thoughts?

I think as you've proposed is great. 👍