Compose recording videos

Encoding
Apparently, you can save a number of video and audio tracks to the same MP4 file:
https://video.stackexchange.com/questions/20933/can-video-have-multiple-streams-like-it-has-2-audio-streams
Streaming
Seems we can also stream directly from the S3 bucket (will probably need to give the player some sort of access token?)
https://stackoverflow.com/a/42176752/6361632
Playback
Option 1:
Use a player that can support displaying multiple channels and/or switching between them.
video.js seems like it may be able to do this: https://videojs.com/guides/video-tracks/, although I don't fully understand what the docs mean.
Option 2: use the HTML5 standard:

https://html.spec.whatwg.org/multipage/media.html#videotrack
https://html.spec.whatwg.org/multipage/media.html#videotrackhttps://stackoverflow.com/questions/52658650/html5-how-to-display-mediastream-with-multiple-video-tracks

Option 3: masking
we could have a different audio track for each participant, and then "mask" parts of the screen if we don't want to show the whole thing.

https://dev.to/benjaminblack/dynamically-masking-video-using-canvas-and-svg-31g9

Encoding - At minimum want to combine each participant's video with their audio file, potentially want to combine all 3 audio/video files into one file so they happen at the same time.

This is the participant data from the recordings i sent you:

It looks like the files themselves are in a directory that matches the room name:

Filenames seem to have 4 parts:
1674660535410-ae4d075b-39b6-48ec-a7c0-868d6fd06afe-cam-audio-1674660543681
1674660535410-ae4d075b-39b6-48ec-a7c0-868d6fd06afe-cam-video-1674660544002

The first section:<PHONE_NUMBER>410 seems to be the UNIX epoch time that converts to the timestamp that the meeting or the first recording started (about 5 seconds after the call started, makes sense)

The next section ae4d075b-39b6-48ec-a7c0-868d6fd06afe seems to correspond to the participant ID
The third section identifies video or audio stream
The last section<PHONE_NUMBER>002 may be the time that the particular track was initialized.

I would guess that if we tried syncing videos based on the last component of the filename, that may be enough. Not entirely sure, so we should test it.

Resources:

https://gingertech.net/2011/05/01/html5-multi-track-audio-or-video/
https://stackoverflow.com/questions/37297856/ffmpeg-combine-video-files-with-different-start-times

Displaying multiple video tracks simultaneously:
Here is an example of a player that has a separate video track for sign language: https://ableplayer.github.io/ableplayer/demos/video5.html
The two videos seem to be included as different sources into a single html5 video component:
https://github.com/ableplayer/ableplayer/blob/b0635e82daf6301c53a8f20b893ee7d2c4d315c2/demos/video5.html#L46-L47

https://stackoverflow.com/questions/52658650/html5-how-to-display-mediastream-with-multiple-video-tracks

Using ffmpeg seems like the best way to combine things into one file
Useful links:
https://superuser.com/questions/277642/how-to-merge-audio-and-video-file-in-ffmpeg
https://blog.addpipe.com/converting-webm-to-mp4-with-ffmpeg/
Command:
ffmpeg -i video.ext -i audio.ext -map 0:v -map 0:a output.mp4
-i video.ext specified first input
-i audio.ext specified second input
-map 0:v maps video streams in first input (index 0) to output.mp4
-map 1:a maps audio streams in second input (index 1) to output.mp4
output.mp4 is the output file name
Notes:
Audio and video clips are synced at the same timestamp based on the start of the call, regardless of when content begins - this means everything can be combined without worrying about coordinating start times
Tried above command - works but tries to slow down a period so that the times match even though they are only off by a few milliseconds (only a problem in QuickTime - can use this command and it works in browser)

I put together a little demo app to explore syncing multiple video streams (at this commit: https://github.com/Watts-Lab/deliberation-video-coding/tree/db3b4952252d04816a96e7a7a4b255b9f4e4cce3)
All I did was use html5 video tags with hardcoded URLs:
...
        <video
          id="myVideo8"
          preload="auto"
          ref={videoElement1}
          width="250"
          controls
        >
          <source
            src="https://wattslab-video-test-public.s3.amazonaws.com/01GR9ED3G57XJBBA77068BNH90/1675354384929-de8e880d-3a37-4149-baba-b3f1aa77cee1-cam-video-1675354437728"
            type="video/webm"
          />
        </video>
...

and attached a react useRef to each.
  const videoElement1 = useRef(null);
  const videoElement2 = useRef(null);
  const videoElement3 = useRef(null);
  ...

Then I could use the refs to start, pause, or seek the videos:
const seekTo = (time) => {
    console.log(`seeking to ${time}`);
    videoElement1.current.currentTime = time;
    videoElement2.current.currentTime = time;
    videoElement3.current.currentTime = time;
   ...
  };

  const startVideos = () => {
    console.log(`starting videos`);
    videoElement1.current.play();
    ...
  };

  const pauseVideos = () => {
    console.log(`starting videos`);
    videoElement1.current.pause();
    ...
  };

This worked surprisingly well, for seeking to times that all videos had in common. The files we have seem to be synchronized so that the "time" in the video file that you seek to is the same for all videos, ie, if you seek to 15 seconds, you get all the videos synced at 15 seconds.
There are a few pieces that didn't work first try, however:

The videos "drift" over time, and go out of sync. This is a known issue with trying to play multiple videos in sync (for example: https://stackoverflow.com/questions/2654817/is-there-a-way-to-keep-two-videos-synchronized-in-a-webpage-without-plugins)
If the video isn't "padded" to cover the full time window (ie, if one video starts later or ends earlier than the others) you can't seek to the right times. Instead, the video will play from its own starting point, which will not be in sync.

We should be able to fix both 1 and 2 with some js in the browser, but it may make sense also to give the videos pre/post padding as needed so that we never try to seek to a part of the conversation that exists in one video but not another. (Assuming we can do this without breaking the timings).

This was in an updated email from daily today:
https://docs.daily.co/reference/vcs
Does it look useful, @kepstein23 ?

Wrote a python script to merge audio and video files, mostly to share with collaborators while we're getting our own system set up.
import json
import glob

filename = "../scienceData/batch_study_2_20230302_prolific_01GTHMZ6YH8X4S5WEMEC06FK77.jsonl"
with open(filename) as f:
    entries = [json.loads(line.rstrip()) for line in f]

basepath = "~/Desktop/DeliberationRecordings"
outputpath = "~/Desktop/FormattedDeliberationRecordings"

i = 0
for entry in entries:
    if 'recordingRoomName' in entry and 'recordingIds' in entry and len(entry['recordingIds'])>0:
        room_name = entry['recordingRoomName']
        recordingIds = entry['recordingIds']
        for recordingId in recordingIds:
            globstring = f'{basepath}/{room_name}/*{recordingId}*'
            files = glob.glob(globstring)
            if len(files) == 2:

                videofile = [file for file in files if "video" in file][0]
                audiofile = [file for file in files if "audio" in file][0]
                outfile = outputpath+"/"+videofile.strip(basepath).replace("cam-video", "merged")+"_"+str(i)+".mp4"
                newdir = "/".join(outfile.split("/")[:-1])

                print("video:", videofile)
                print("audio:", audiofile)
                print("outfile", outfile)
                print("newDir", newdir)
                print("")

                !mkdir {newdir}
                !ffmpeg -i {videofile} -i {audiofile} -map 0:v -map 1:a {outfile}
                i += 1

I think we decided not to do this for now, and just do real-time stitching in the interface.


Wire Launch

Preface: This is not an issue, but I didn't think the Google groups were the right venue for this discussion. Feel free to punt this elsewhere, etc.
Hey team! We at Bolt are definitely intrigued by this project, and we're sort of charting our migration path forward before we embark on the development work. The large outstanding question at this point is when this project will make it out of alpha, and I'm curious how you're thinking about timelines/the maturity of this project. Are there any timeline goals for the project?
While we'd love to adopt this framework, Facebook's injection is working generally fine for us. Therefore, we can't quite make the case just yet to adopt without it coming out of alpha, but when it does, we'd definitely want to give it a shot.
Thanks! And great work!

Hi, I have the same issue. I could use Wire successfully in few projects. But I am finding difficult to explain the alpha status of wire to my team.

Hey folks! Happy to hear the interest. At the moment, we have a few outstanding issues that we want to address before we graduate Wire to beta:

Changes to struct providers: #36, #32, and #110
**Fixing an inconsistency in wire.Bind: #120

Once we've fixed those, we are fairly confident we can make future changes in a backward-compatible manner. These changes are fairly minimal, and hopefully there's enough information in those issues that your organizations can plan accordingly. Thus, we would declare Wire as beta and then graduate to 1.0 for a period of time after that.
I don't have an exact timeline we can communicate at the moment, but I would expect beta to come in the next month or so, and 1.0 sometime before end of 2019. Does this address your question?

@zombiezen Thanks! I appreciate the response, and the transparency. This does address my questions, but I do have one last question.
While I'm sure you can't answer with certainty, is this a project/framework that Google is committed to maintaining in the long run? Is there any dependency on framework usage in the beta version to determine if this will move to 1.0?

We are not using usage as a measure of 1.0-ness; we are using stability of API as a measure of 1.0-ness. We have every intention of continuing to maintain Wire, as it is a critical piece of infrastructure for the Go CDK.

Got it - great to know! I'm closing this out and watching the releases. In due time, when the beta comes out, I'm sure we'd love to incorporate and contribute as well.
Thanks.

Hey @shannontan-bolt and @h7kanna! If you didn't see, we announced Wire as beta last week. Please file issues if you encounter any issues. Cheers!

@zombiezen - thanks so much!


jest-dev-server isn't waiting to finish my bundle

🐛 Bug Report
I want to run the tests on my CI, so I need to start the webpack-dev-server before to run the tests.
So jest-dev-server would be very good to me, but I noticed that it isn't waiting my bundler finish before to run the tests.
My bundle process is long: it takes ~16 seconds to be able to see the index.
But since jest-dev-server just wait the port be available, it will fail on tests, because the port is open much before to server be ready.
To Reproduce
Just have a very complex build process to see this bug.
Also, if start manually the server before to run the tests, it'll work. But is bad to write a code like that to run automatically on CI.
Expected behavior
jest-dev-server should wait the build finish before to start (maybe afterCompile could help).

I really don't know if it's the solution, but I solved that setting to wait-on wait to /webpack-dev-server be available instead of the port:
  server: {
    command: 'webpack-dev-server --mode development --env=local --config webpack.config.js',
    port: 8080,
    debug: true,
    launchTimeout: 30000,
    waitOnScheme: {
      verbose: true,
      resources: ['http://localhost:8080/webpack-dev-server'],
    },
  },

I think that it could be the default behaviour. Wait for /webpack-dev-server instead of just the tcp:localhost:8080

@macabeus I also experienced this problem, but solved it differently. I noticed that by default, jest-dev-server calls  wait-on with an URL that starts with http:// or https://, see here https://github.com/smooth-code/jest-puppeteer/blob/0d72f09fa95173bf63999e05368bd4f312cde851/packages/jest-dev-server/src/global.js#L195
This in turn means it uses HTTP HEAD to call the resources
       http:      - HTTP HEAD returns 2XX response. ex: http://m.com:90/foo
       https:     - HTTPS HEAD returns 2XX response. ex: https://my/bar

Using HEAD, in an older version of webpack-dev-server that I had, it replied before the bundle finished. Trying a GET request it properly replied after the bundle had finished. I found out that this was due to that webpack-dev-server had no proper HEAD support until verison 3.4.0, see https://github.com/webpack/webpack-dev-server/releases/tag/v3.4.0

@macabeus solution worked for me 👍

You should do the bundle before you start the tests.
// in package.json
  "scripts": {
    "pretest": "webpack-dev-server --mode development --env=local --config webpack.config.js"
  }


Added Slider Layout

Resolves #153

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 98.49%. Comparing base (838c8b4) to head (ca5ec75).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #154      +/-   ##
==========================================
+ Coverage   98.40%   98.49%   +0.09%
==========================================
  Files          55       59       +4
  Lines        1193     1265      +72
  Branches      342      365      +23
==========================================
+ Hits         1174     1246      +72
+ Misses         19       18       -1
- Partials        0        1       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


IndexSizeError in IE11 with SVG,PNG on VerctorLayer

I'm currently using a combination of SVG and PNG icons in my Map.
I need to use the PNGs as IE11 is not able to colorize the SVGs correctly. But  i get an IndexSizeError when the images are to be drawn or redrawn.
I have prepared a plunkr that shows the error when one clicks on the Icons in IE11. When the getFeaturesUnderPixel() initiates replayHitDetection() and the redraw is called we get the error.
In my example I have linked Version 3.16.0 (which i'm currently using) but the error also occurrs in master branch.
This issue does not occurr in Chrome or Firefox browsers.

See https://github.com/openlayers/openlayers/issues/6281#issuecomment-268962325, closing as duplicate.


Is this broken on the latest Webui?

Pressing send just hangs on "[LPP] Fetching tags: 0it [00:00, ?it/s]" or am i dumb?

Connectivity issues, most likely. Note that e621/derpi are blocked in some countries. Is the website you're trying to fetch tags from opens normally for you in a browser?

Derpibooru, it loads fine and I know my API key works with other apps.

nevermind.


Use package imports instead of path alias.

Have to wait for typescript to solve finally. Partially solved by a5c42a6fb6f9e01


loadImage

When running the Run function in the loadImage function, it is always slow, is there any solution? Is the process a must every time a new picture is loaded? Can it be reused to reduce pre-treatment time?

The only solution is to have a good gpu and run with cuda device. If you are running it on cpu, I'm not surprised. Image embedding is not real time (cf : paper of sam) !


TopNavigationAction is not show the back icon on UI Kitten 5

🐛 Bug Report
I just tried the new version 5.0.0. I followed the example of how to implement the TopNavigationAction. the icon on top is not shown.
I did not install the font, I guess this has nothing to do with that.
to confirm that the icon component is working in general, on the screen I added the Facebook icon without an issue. only on the nav. BTW, the same issue happened on both iOS and Android

To Reproduce
I used expo to install all required libs following the steps below:
https://akveo.github.io/react-native-ui-kitten/docs/guides/getting-started#getting-started
https://akveo.github.io/react-native-ui-kitten/docs/guides/icon-packages#icon-packages
https://akveo.github.io/react-native-ui-kitten/docs/guides/configure-navigation#configure-navigation
Expected behavior
I expect to see the back icon and make it function on both Android and iOS
If I tried to install expo install<EMAIL_ADDRESS>I got this version in package.json
"react-native-svg": "11.0.1",
I tried to change the version in package.json to 9.13.6 then re-run the npm I but it failed in building the app.

UI Kitten and Eva version
|"dependencies": {
"@eva-design/eva": "^2.0.0-alpha.1",
"@react-native-community/masked-view": "0.1.6",
"@react-navigation/native": "^5.1.5",
"@react-navigation/stack": "^5.2.10",
"@ui-kitten/components": "^5.0.0-alpha.1",
"@ui-kitten/eva-icons": "^5.0.0-alpha.1",
"expo": "~37.0.3",
"react": "~16.9.0",
"react-dom": "~16.9.0",
"react-native": "https://github.com/expo/react-native/archive/sdk-37.0.1.tar.gz",
"react-native-gesture-handler": "~1.6.0",
"react-native-reanimated": "~1.7.0",
"react-native-safe-area-context": "0.7.3",
"react-native-screens": "~2.2.0",
"react-native-svg": "11.0.1",
"react-native-web": "~0.11.7"
},
"devDependencies": {
"babel-preset-expo": "~8.1.0",
"@babel/core": "^7.8.6"
},
Environment information

System:
OS: macOS 10.15.4
CPU: (8) x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Memory: 356.41 MB / 16.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 12.16.1 - /usr/local/bin/node
npm: 6.13.4 - /usr/local/bin/npm
Managers:
CocoaPods: 1.0.1 - /Users/mohammed-macbookPro/.rvm/rubies/ruby-2.3.0/bin/pod
Homebrew: 2.1.11 - /usr/local/bin/brew
pip3: 19.0.3 - /usr/bin/pip3
RubyGems: 2.5.1 - /Users/mohammed-macbookPro/.rvm/rubies/ruby-2.3.0/bin/gem
Utilities:
Make: 3.81 - /usr/bin/make
GCC: 4.2.1 - /usr/bin/gcc
Git: 2.23.0 - /usr/local/bin/git
Clang: 1<IP_ADDRESS> - /usr/bin/clang
Servers:
Apache: 2.4.41 - /usr/sbin/apachectl
Virtualization:
Parallels: 15.1.3 - /usr/local/bin/prlctl
SDKs:
iOS SDK:
Platforms: iOS 13.4, DriverKit 19.0, macOS 10.15, tvOS 13.4, watchOS 6.2
Android SDK:
API Levels: 17, 18, 19, 20, 21, 22, 23, 26, 27, 28, 29
Build Tools: 29.0.3
System Images: android-23 | Google APIs Intel x86 Atom, android-29 | Google APIs Intel x86 Atom
IDEs:
Android Studio: 3.6 AI-192.7<IP_ADDRESS>41897
Nano: 2.0.6 - /usr/bin/nano
Vim: 8.1 - /usr/bin/vim
WebStorm: 2019.3.3
Xcode: 11.4/11E146 - /usr/bin/xcodebuild
Languages:
Bash: 3.2.57 - /bin/bash
Java: 1.8.0_101 - /usr/bin/javac
Perl: 5.18.4 - /usr/bin/perl
PHP: 7.3.11 - /usr/bin/php
Python: 2.7.16 - /usr/bin/python
Python3: 3.7.3 - /usr/bin/python3
Ruby: 2.3.0 - /Users/mohammed-macbookPro/.rvm/rubies/ruby-2.3.0/bin/ruby
Databases:
MongoDB: 4.2.0 - /usr/local/bin/mongo
SQLite: 3.28.0 - /Users/mohammed-macbookPro/Library/Android/sdk/platform-tools/sqlite3
Browsers:
Chrome: 80.0.3987.163
Firefox: 66.0.2
Safari: 13.1

I suppose you're doing something wrong. See the working example on snack https://snack.expo.io/WcuOHWMqE

thank you Artur,
it was a documentation issue. in the example.
it used leftControl={BackAction()} whereas it should use accessoryLeft={BackAction}
<TopNavigation title='MyApp' alignment='center' leftControl={BackAction()}/>
<TopNavigation  title='MyApp' alignment='center' accessoryLeft={BackAction}/>
Thanks

@mhhonline

it was a documentation issue. in the example.

Was it? Can't see there is a leftControl prop in the examples

@artyorsh
you are right, in the component example's is correct.
I started from the beginning here in "Details Screen"
https://akveo.github.io/react-native-ui-kitten/docs/guides/configure-navigation#create-screens
you can see leftControl
return (
    <SafeAreaView style={{ flex: 1 }}>
      <TopNavigation title='MyApp' alignment='center' leftControl={BackAction()}/>
      <Divider/>
      <Layout style={{ flex: 1, justifyContent: 'center', alignItems: 'center' }}>
        <Text category='h1'>DETAILS</Text>
      </Layout>
    </SafeAreaView>
  );


Remove _document and _app boilerplate

For some reason I sometimes needed _document for styled-components to work. But after removing it things are working fine for me. Also we weren't doing anything special with _app so I removed that too.
@dwalkr do you know of any reason we need either of those for this kind of architecture?

@jeffsee55 _app is just used to ensure TinaProvider is added to all routes, but if you don't actually want that then it isn't necessary. The _document code to make styled-components work is still necessary as far as I know, but isn't always an issue for dev mode (only in the production build.) If that is truly no longer needed, then I'd assume something was fixed upstream in Next.js

Got it, thanks. I'm going to close this for now and circle back on cleanup if Nextjs did do some upstream improvements but not that I can see in their releases. It was working for me but I'm guessing that's not the case for prod builds


CI: do not hold kubernetes packages

There is no need to hold the kubernetes packages to
a specific version in the CI. Holding them was preventing
the metrics CI node to fail installing the correct packages.
Fixes: #950
Signed-off-by: Salvador Fuentes<EMAIL_ADDRESS>

This looks fine. But I wonder if we need to special-case the metrics CI so that only it removes the hold) rather than removing the hold entirely?
/cc @grahamwhaley.

Right now the metrics CI doesn't do any k8s testing (yet...), so the change should not affect the present restults.
What I don't quite understand is why the metrics CI was failing to install packages, but presumably the QA CI was not - I'm guessing different versions of Ubuntu maybe? Do we have a few more details on this?

lgtm

@grahamwhaley It started to fail when we moved k8s from v1.9.2 to v1.9.3. As apt was holding the version of the packages, they couldn't be upgraded and the CI failed on this step. And as on QA CI, every run is a new VM, the problem didnt appear.


There is a problem with module procedure <blas_routine>

It seems I need to explicitly define the F77 to each interface in order to use a "type agnostic" F77 BLAS function
~$ fypp blas.fpp blas.90
~$ gfortran blas.f90 /usr/lib/libblas.a

blas.f90:77:21:

   77 |     module procedure dzasum
      |                     1
Error: Procedure ‘dzasum’ in generic interface 'f77_asum' at (1) is neither function nor subroutine
blas.f90:51:21:

   51 |     module procedure zaxpy
      |                     1
Error: Procedure ‘zaxpy’ in generic interface 'f77_axpy' at (1) is neither function nor subroutine
blas.f90:259:21:

This can be solved by removing the pure atribute to each new interface and calling the BLAS/LAPACK procedure directly.
MKL defines their own pure F77 interfaces to BLAS/LAPACK procedures and then call it in their F95 interface.
LAPACK95 also defines their own F77 interfaces but without the pure attribute.
I don't know what is the difference... I think I will stick with my current approach by making the modern interface impure procedures and then calling the F77 version.

Including my own pure F77 interfaces solves the issue, everything is pure now.


Dialog tab order

You can tab from "confirm" to "cancel" button in dialogs but cannot tab through "close" button. Also, tab order doesn't wrap e.g. when you get to the end of the tab sequence, it stops tabbing although you can go back the way using Shift+Tab.
These have been flagged by our QA team as accessibility issues.
Steps to reproduce the behaviour:

Go to Radzen Confirm Dialog
Click on "Confirm dialog" button
Try tabbing
Observe behaviour

Expected behaviour is being able to cycle focus through close, confirm and cancel buttons in the dialog using the tab button (as is the case with blazorise modal - see Blazorise Modal component

I'll push update on the focus logic to allow cycle between first and last elements in the dialog content however this will not include the elements in the title (the close button is there), still I've enabled ENTER and SPACE keys to be used to close the dialog if pressed over close button. You can close the dialog using ESC key.

Thank you!

Any future plans to enable cycling through the close button in the title as well as the dialog content elements?

We will try to enable this however we are not sure if we will be able.


[Missing SNMP Polling Profile] - ArubaS2500-48P-US

Device Vendor:  Aruba
Device Model: ArubaS2500-48P-US
System Object Identifier (SysOID): <IP_ADDRESS>.4.1.148<IP_ADDRESS>
List of critical metrics that you expect in the profile (does not need to be all-inclusive, just a general idea of your expectations):

Total Bandwidth
-Traffic IN/OUT
-Interface UP/DOWN
Uptime
CPU Utilization
VLAN Bandwidth

Please provide a public gist with a sanitized (sensitive information removed) SNMP walk of the device. Providing MIB files is not a substitute for an SNMP walk and failure to provide one here will result in delays for closing this issue. The output of these commands shows the "true story" of what a device will respond to in SNMP and is used to ensure that our profiles are efficient and accurate for your devices.
Gist Link:
https://gist.github.com/ThromboP/5b724adab656739932a5d87d522c554c

Example SNMP walk commands:
# v2c example
snmpwalk -v 2c -On -c $COMMUNITY_STRING $IP_ADDRESS . >> snmp_walk.out

# v3 example
snmpwalk -v3 -l authPriv -u $USERNAME -a MD5|SHA -A $AUTH_PASS -x DES|AES -X $PRIV_PASS -ObentU -Cc $IP_ADDRESS . >> snmp_walk.out

resolved with #197


Implements SFML.Window.Event.TextEvent.

Uses this to fix TextBoxes.
Also fixes the console being unable to be opened, new key is Home.
KNOWN BUG: bound keys still happen, even if textboxes are focused.

I hate how we propagate events, but the PR looks good nevertheless.

Yeah it's pretty bad but it's very modular.
Could use some unspaghettification though.


Mac: Matches up KextLog levels with os_log levels

The existing levels in the KextLog mechanism have repeatedly caused confusion with regard to their relative importance and whether or not they hit the on-disk system log.

The "Note" level has been replaced with "Default" to match its os_log counterpart. The corresponding macros have been renamed to KextLog() and KextLog_File(), so "Default" need not be spelled out explicitly.
Comments have been added to the log level enum definition to explain the syslog policy.

Looks like the Windows functional tests hit #380, I will kick them off again


Export Logs creates an empty file for the first attempt

When I use the Export Logs feature, it gives me an empty file. It works when I do the same action again for the second time.

There appears to be a bit of a race condition going on. The Export Logs sets the DittoLogger.setLogFileURL, which will Register a file path where logs will be written to, whenever Ditto wants to issue a log (on top of emitting the log to the console). Once that is set then that log file is read from and the zip file is created. However, there is a delay from when the setLogFileURL is set to when Ditto starts writing to the file. One solution would be to move the  DittoLogger.setLogFileURL(logFileURL) part out of the Export Logs and add it to the app logic when setting the DittoLogger.minimumLogLevel. This gives Ditto enough time to start writing the logs to the file, but it also means developers will need to make sure they set the DittoLogger.setLogFileURL.


working DistilBERT example with notebook

Working DistilBERT fine tuning example with DP-SGD (on the SMS Spam Collection) and accompanying notebook

Also, regarding the notebook, ideally, you would want the user to install from Pypi right?
So, instead of
cd bastionai/client && make install

you would want to do
! pip install bastionai

Exactly why not having a Makefile is a good idea, and it simplifies installing the package.
Once you have a setup.py file, it setups up all the necessary configuration when you run pip install bastionai

Nice LICENSE file. I want to ask why you want to use a Makefile instead of setup.py and add all the necessary build steps.
Just like it is now?

So, I think the new standard to build python packages is to use setup.cfg + pyproject.toml instead of a raw setup.py, that's why I moved around some of the build configs
The Makefile is a good way to reproduce the dev environement and to ease building by writting all the steps at the same place. It is widely used, installed on all Linux platforms, easy to understand and adapt.

Also, I noticed you added an examples directory under client.
How about putting the bert in the root of the project, adding it unto SampleTraining.ipynb?

That's actually a good idea, I also think that all current files under tests and examples should be moved to the root as they use both the client and the server. We can them make unit tests for each component

Also, regarding the notebook, ideally, you would want the user to install from Pypi right? So, instead of
cd bastionai/client && make install

you would want to do
! pip install bastionai

Exactly why not having a Makefile is a good idea, and it simplifies installing the package. Once you have a setup.py file, it setups up all the necessary configurations when you run `pip install bastionai
And for the extra pip packages needed in the notebook, I believe you can easily install them in the notebook by running
! pip install transformers pandas sklearn ipykernel ipywidgets

This is a very good point
I'll fix the notebook and conform to the ! syntax
Regarding the bastionai pip package, it is not ready yet so I would jut include the two installation schemes (package manager + from source)


lightgbm.basic.LightGBMError: Label 72 is not less than the number of label mappings (31)

Hello to everyone!!
I am new to Python and Iam getting this error when running LightGBM about a Ranking problem: lightgbm.basic.LightGBMError: Label 72 is not less than the number of label mappings (31)
I tried to search for this error, could not find much useful resources.
I cant guess where the error occurs.
Μy dataset consists of 4 columns: ["Frequency","Comments", "Likes", "Nwords"] as seen below.
# 1) Load Dependencies
import pandas as pd
from numpy import where
import matplotlib.pyplot as plt
import numpy as np
from numpy import unique
from sklearn import metrics
import warnings
warnings.filterwarnings('ignore')
import lightgbm as lgb
gbm = lgb.LGBMRanker()

# 2) Load the Data
# Define Columns
names = ["Frequency","Comments", "Likes", "Nwords"]]

data = pd.read_csv("Posts.csv", encoding="utf-8", sep=";", delimiter=None,
                 names=names, delim_whitespace=False,
                 nrows=181,header=0, engine="python")
X = data.values[:,0:2]
y = data.values[:,3]

# 3) Define the Training Data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=1)

query_train = [X_train.shape[0]]
query_val = [X_val.shape[0]]
query_test = [X_test.shape[0]]
# label_values = [label_gain(1,5)]

gbm.fit(X_train, y_train, group=query_train,
        eval_set=[(X_val, y_val)], eval_group=[query_val], #values=[label_gain(1,5)],
        eval_metric=["ndcg"],eval_at=[1, 2, 3,4,5], early_stopping_rounds=10)

# make predictions
test_pred = gbm.predict(X_test)
X_test["predicted_ranking"] = test_pred
X_test.sort_values("predicted_ranking", ascending=False)

Can anyone help me??
Thank you in advance !!
Sofia

Hi @sofiavlachou28 , thanks very much for using LightGBM!
I'd be happy to help you, but we need a little more information.

What version of lightgbm are you using and how did you install it?
Are you able to provide access to the raw data ("Posts.csv"), or to replicate this problem using randomly-created data?

@sofiavlachou28 please refer to parameter label_gain

Hi @sofiavlachou28 , thanks very much for using LightGBM!
I'd be happy to help you, but we need a little more information.

What version of lightgbm are you using and how did you install it?
Are you able to provide access to the raw data ("Posts.csv"), or to replicate this problem using randomly-created data?

@jameslamb I use the Version 3.3.1. Also, I uploaded my csv dataset for more understaning.
Posts.csv
Thanks for your time!! :)

@jameslamb Also, I saw that lightgbm requires:  numpy, scipy, scikit-learn, wheel. I use the latest versions of numpy, scipy, sklearn. But when I try to upgrade the "weel" a new error occures:
(venv) C:\Users\USER\pythonProject>pip install wheel==0.37.0
Collecting wheel==0.37.0
Downloading wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Attempting uninstall: wheel
Found existing installation: wheel 0.36.2
Not uninstalling wheel at c:\users\user\appdata\local\programs\python\python37\lib\site-packages, outside environment c:\users\user\pythonproject\venv
Can't uninstall 'wheel'. No files were found to uninstall.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following de
pendency conflicts.
tf-nightly-gpu 2.5.0.dev20210223 requires gast==0.4.0, but you have gast 0.3.3 which is incompatible.
tf-nightly-gpu 2.5.0.dev20210223 requires grpcio~=1.34.0, but you have grpcio 1.32.0 which is incompatible.
tf-nightly-gpu 2.5.0.dev20210223 requires h5py~=3.1.0, but you have h5py 2.10.0 which is incompatible.
tf-nightly-gpu 2.5.0.dev20210223 requires six~=1.15.0, but you have six 1.16.0 which is incompatible.
tf-nightly-gpu 2.5.0.dev20210223 requires typing-extensions~=3.7.4, but you have typing-extensions <IP_ADDRESS> which is incompatible.
tensorflow 2.4.1 requires six~=1.15.0, but you have six 1.16.0 which is incompatible.
tensorflow 2.4.1 requires typing-extensions~=3.7.4, but you have typing-extensions <IP_ADDRESS> which is incompatible.
tensorflow-gpu 2.4.1 requires six~=1.15.0, but you have six 1.16.0 which is incompatible.
tensorflow-gpu 2.4.1 requires typing-extensions~=3.7.4, but you have typing-extensions <IP_ADDRESS> which is incompatible.
Successfully installed wheel-0.37.0
(venv) C:\Users\USER\pythonProject>

I dont understand what happens again...
Any idea?
Thanks a lot..!

@sofiavlachou28 Thanks for using LightGBM! Ranking objectives in LightGBM use label_gain_ to store the gain of each label value. By default, label_gain_[i] = (1 << i) - 1. So the default label gain only works with a maximum label value 31. It seems that your dataset contains label values greater than 31. So please specify your customized label_gain, as @guolinke mentioned.

@sofiavlachou28 Thanks for using LightGBM! Ranking objectives in LightGBM use label_gain_ to store the gain of each label value. By default, label_gain_[i] = (1 << i) - 1. So the default label gain only works with a maximum label value 31. It seems that your dataset contains label values greater than 31. So please specify your customized label_gain, as @guolinke mentioned.

Okay. I will try it with this parameter. I hope it works..!
Τhanks for your help!

Thanks for providing the dataset @sofiavlachou28 .
Looking at the data, I have some observations and a suggestion.
The label for a learning-to-rank problem is expected to be a "relevance score", explaining how relevant one document is compared to another. (see this Stack Overflow answer for a concise explanation).
If you set  label_gain to the maximum value in y_train as suggested in https://github.com/microsoft/LightGBM/issues/4808#issuecomment-973991963, model training might run without throwing any errors, but (making some assumptions about your data based on column names), I don't think the model generated will be what you intended.
It seems you're using column nwords for the label, which I assume is "number of words in the post". If you want to use LightGBM to predict the number of words in a document based on how popular it was (likes, comments), I recommend treating that as a regression problem and using LGBMRegressor, not LGBMRanker.
One other suggestion...I noticed the full dataset has only 92 rows, even before holding out some data for validation. LightGBM has a few parameters to limit model complexity, whose defaults are set to work well with medium-sized datasets (1000s of observations). If you want LightGBM to learn from 92 observations, consider setting a very small value (like 2) for parameter min_data_in_leaf (link).

Thanks for providing the dataset @sofiavlachou28 .
Looking at the data, I have some observations and a suggestion.
The label for a learning-to-rank problem is expected to be a "relevance score", explaining how relevant one document is compared to another. (see this Stack Overflow answer for a concise explanation).
If you set label_gain to the maximum value in y_train as suggested in #4808 (comment), model training might run without throwing any errors, but (making some assumptions about your data based on column names), I don't think the model generated will be what you intended.
It seems you're using column nwords for the label, which I assume is "number of words in the post". If you want to use LightGBM to predict the number of words in a document based on how popular it was (likes, comments), I recommend treating that as a regression problem and using LGBMRegressor, not LGBMRanker.
One other suggestion...I noticed the full dataset has only 91 rows, even before holding out some data for validation.
sample code (click me)
LightGBM has a few parameters to limit model complexity, whose defaults are set to work well with medium-sized datasets (1000s of observations). If you want LightGBM to learn from 92 observations, consider setting a very small value (like 2) for parameter min_data_in_leaf (link).

Τhank you so much for your reply! it is very helpful! I will look at your observations more carefully and I will test my data again, as you suggest. I hope it works. If not, I will open the topic again!
Have a good day!
Sofia

Hello to everyone!
After some research and a few hours of hacking, my code is still not working.  I do not find anywhere information on how to set the label_gain as you suggested above!
My task is to find the most popular product each time based on likes, comments, or frequency and so on.
Here is my code!       I am new to Python ! Can anyone help me??
# 1) Load Dependencies
import pandas as pd
import numpy as np
from numpy import unique
from sklearn import metrics
import warnings
warnings.filterwarnings('ignore')

import lightgbm as lgb

# Core Model
gbm = lgb.LGBMRanker(objective="lambdarank", )

# 2) Load the Data
# Define Columns
names = ["memespostsfrequency","comments","likes","nwords"]

data = pd.read_csv("InstaPosts.csv", encoding="utf-8", sep=";", delimiter=None,
                 names=names, delim_whitespace=False,
                 nrows=181,header=0, engine="python")
X = data.values[:,0:2]
y = data.values[:,3]

# 3) Define the Training Data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=1)

query_train = [X_train.shape[0]]
query_val = [X_val.shape[0]]
query_test = [X_test.shape[0]]

# 4) Model Fit
gbm.fit(X_train,
        y_train,
        group=query_train,
        eval_set=[(X_val, y_val)],
        eval_group=[query_val], #values=[label_gain(1,5)],
        eval_metric=["ndcg"],
        eval_at=[3],
        early_stopping_rounds=10)

# 5) Predictions
test_pred = gbm.predict(X_test)
X_test["predicted_ranking"] = test_pred
X_test.sort_values("predicted_ranking", ascending=False)

And here is the bug:
C:\Users\USER\pythonProject\venv\Scripts\python.exe "C:/Users/USER/pythonProject/Memes LightGBM Algorithm.py"
[LightGBM] [Fatal] Label 72 is not less than the number of label mappings (31)
Traceback (most recent call last):
  File "C:/Users/USER/pythonProject/Memes LightGBM Algorithm.py", line 87, in <module>
    early_stopping_rounds=10)
  File "C:\Users\USER\pythonProject\venv\lib\site-packages\lightgbm\sklearn.py", line 1071, in fit
    categorical_feature=categorical_feature, callbacks=callbacks, init_model=init_model)
  File "C:\Users\USER\pythonProject\venv\lib\site-packages\lightgbm\sklearn.py", line 758, in fit
    callbacks=callbacks
  File "C:\Users\USER\pythonProject\venv\lib\site-packages\lightgbm\engine.py", line 271, in train
    booster = Booster(params=params, train_set=train_set)
  File "C:\Users\USER\pythonProject\venv\lib\site-packages\lightgbm\basic.py", line 2613, in __init__
    ctypes.byref(self.handle)))
  File "C:\Users\USER\pythonProject\venv\lib\site-packages\lightgbm\basic.py", line 125, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: Label 72 is not less than the number of label mappings (31)

Process finished with exit code 1

Thank you in advance!!
Sofia V.

I had the same problem with label mappings when I tried to use LGBMRanker with Optuna. I did them work well together by following this example. As suggested, I set the label_gain parameter as:

[i for i in range(max(y_train.max(), y_valid.max()) + 1)]

I hope it can also help you!
Best,

I had the same problem with label mappings when I tried to use LGBMRanker with Optuna. I did them work well together by following this example. As suggested, I set the label_gain parameter as:

[i for i in range(max(y_train.max(), y_valid.max()) + 1)]

I hope it can also help you!
Best,

@thaisalmeida Thanks for your reply!
I don't understand how to set the label_gain in my code... still raises an error: lightgbm.basic.LightGBMError: Label 47 is not less than the number of label mappings (31)
Can share your code ? or if you don't want, can you help me to set this parameter in my code below?
Thank you in advance :) !!
Here is my New code:
# Dependencies
import pandas as pd
from pandas import set_option
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import RepeatedStratifiedKFold
from pandas import read_csv
import numpy as np
from numpy import unique
from sklearn import metrics
# LGBMRanker
import lightgbm as lgb
from lightgbm import LGBMRanker
from sklearn.model_selection import train_test_split
import warnings
warnings.filterwarnings('ignore')

# Load data
names = ["label","id","comments","likes","product 1 frequency","product 2 frequency"]
dataset = pd.read_csv("Ranking problem.csv", names=names, encoding="utf-8", error_bad_lines=True,
					   skip_blank_lines=True, sep=",", delimiter=None, doublequote=True, keep_default_na=True,
					   nrows=1223, header=6, engine="python")

# Shape
print(dataset.shape)

# Max labels
max_label = dataset.label.nunique()
print(max_label)

# Core Model
gbm = lgb.LGBMRanker(objective="lambdarank", )

# label_gain=np.arange(1, max_label+1)

# Split the data in train and test
array = dataset.values
X = array [:,0:4]
y = array [:,5]
X = X.astype('int64')
y = LabelEncoder().fit_transform(y.astype('str'))

X_train, X_test, y_train, y_test = train_test_split(X,y,
                                                    train_size=0.75, test_size=0.25,random_state=42)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)

# define search
model = lgb.LGBMRanker()

# perform the search
model.fit(X, y,
		  group=[400,400,416])

# 5) Predictions
test_pred = gbm.predict(X_test)
X_test["predicted_ranking"] = test_pred
X_test.sort_values("predicted_ranking", ascending=False)

@sofiavlachou28 Thanks for your interest in LightGBM!
I wrote up a learning-to-rank example tonight to hopefully answer this and other issues you've opened regarding LGBMRanker in the Python package (#5297, #5283).

label_gain
As described in https://lightgbm.readthedocs.io/en/latest/Parameters.html#objective:

label_gain can be used to set the gain (weight) of int label and all values in label must be smaller than number of elements in label_gain

And as described in https://lightgbm.readthedocs.io/en/latest/Parameters.html#label_gain

...only used in lambdarank application

group parameter
As described in https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRanker.html#lightgbm.LGBMRanker.fit

Group/query data. Only used in the learning-to-rank task. sum(group) = n_samples. For example, if you have a 100-document dataset with group = [10, 20, 40, 10, 10, 10], that means that you have 6 groups, where the first 10 records are in the first group, records 11-30 are in the second group, records 31-70 are in the third group, etc.

This parameter is necessary to tell LightGBM which collections of rows in the training data represent documents from the same "query". If you aren't literally working with search engine data (where you have a list of results return by a single search), you might define "query" as, for example, "all movie ratings created by one user".

Sample Code
I created the following example using Python 3.8.12 and lightgbm installed from source from the latest commit on master (https://github.com/microsoft/LightGBM/commit/9489f878b3568e70b441e5df602483e116f24cc6).
The example below uses LightGBM to build a learning-to-rank model to learn how users in the MovieLens-100K dataset rated different movies.
import io
import os
import zipfile
import lightgbm as lgb
import pandas as pd
import requests
from scipy.stats import spearmanr

def load_movielens(local_dir) -> pd.DataFrame:
    data_url = "https://files.grouplens.org/datasets/movielens/ml-100k.zip"
    if not os.path.isdir(local_dir):
        print(f"creating directory '{local_dir}' to store movielens dataset")
        os.mkdir(local_dir)

    zip_file = zipfile.ZipFile(io.BytesIO(requests.get(data_url).content), "r")

    zip_file.extract(
        member="ml-100k/u.data",
        path="data/"
    )
    rating_df = pd.read_csv(
        "data/ml-100k/u.data",
        sep="\t",
        header=None,
        names=["user_id", "item_id", "rating", "timestamp"]
    )
    zip_file.extract(
        member="ml-100k/u.user",
        path="data/"
    )
    user_df = pd.read_csv(
        "data/ml-100k/u.user",
        sep="|",
        encoding="latin-1",
        header=None,
        names=["user_id", "age", "gender", "occupation", "zip_code"]
    )
    zip_file.extract(
        member="ml-100k/u.item",
        path="data/"
    )
    item_df = pd.read_csv(
        "data/ml-100k/u.item",
        sep="|",
        encoding="latin-1",
        header=None,
        names=[
            "movie_id",
            "movie_title",
            "release_date",
            "video_release_date",
            "imdb_url",
            "genre=unknown",
            "genre=Action",
            "genre=Adventure",
            "genre=Animation",
            "genre=Childrens",
            "genre=Comedy",
            "genre=Crime",
            "genre=Documentary",
            "genre=Drama",
            "genre=Fantasy",
            "genre=Film_Noir",
            "genre=Horror",
            "genre=Musical",
            "genre=Mystery",
            "genre=Romance",
            "genre=Sci_Fi",
            "genre=Thriller",
            "genre=War",
            "genre=Western"
        ]
    )
    out_df = rating_df.merge(
        right=user_df,
        how="left",
        on=["user_id"],
        suffixes=("_rating", "_user")
    )
    out_df = out_df.merge(
        right=item_df,
        how="left",
        left_on=["item_id"],
        right_on=["movie_id"],
        suffixes=(None, "_movie")
    )
    # drop join keys and other unnecessary columns
    out_df.drop(["imdb_url", "item_id", "movie_id", "movie_title", "video_release_date", "zip_code"], axis=1, inplace=True)
    out_df = out_df.sort_values(["user_id"], ignore_index=True)

    # LightGBM assumes rankings begin at 0, but these ratings go from 1 to 5
    rating = out_df.pop("rating").values - 1

    # use "user_id" to group queries
    user_id = out_df.pop("user_id")
    group = user_id.value_counts(sort=False).values

    return out_df, rating, group

# get movielens data
X, y, g = load_movielens("data")

# collapse 1-hot-encoded genre into 1 feature
genre_columns = [c for c in X.columns if c.startswith("genre")]
X["movie_genre"] = X[genre_columns].head().idxmax(1)
X.drop(genre_columns, axis=1, inplace=True)

# create a "movie age" feature
X["movie_age_when_rated"] = (
    pd.to_datetime(X["timestamp"], unit="s") -
    pd.to_datetime(X["release_date"])
) / pd.Timedelta(days=1)
X.drop(["timestamp", "release_date"], axis=1, inplace=True)

# convert "object" columns to unordered categories
for col in X.columns:
    if pd.api.types.is_object_dtype(X[col]):
        X[col] = pd.Categorical(X[col])

Looking at the shape of these objects may be information.
The features include some characteristics of the reviewer and some characteristics of the movies.
print(X.head().to_markdown())

|    |   age | gender   | occupation   | movie_genre   |   movie_age_when_rated |
|---:|------:|:---------|:-------------|:--------------|-----------------------:|
|  0 |    24 | M        | technician   | genre=Crime   |               1362.16  |
|  1 |    24 | M        | technician   | genre=Western |               2133.31  |
|  2 |    24 | M        | technician   | genre=Action  |               6841.15  |
|  3 |    24 | M        | technician   | genre=Comedy  |                362.215 |
|  4 |    24 | M        | technician   | genre=Action  |               1362.16  |

The target is integer ratings from 0 to 4 (where 0 is very bad and 4 is very good).
y[:10]
# array([4, 3, 4, 4, 3, 2, 3, 3, 3, 3])

And group groups all ratings from one user together as one "query".
g[:10]
# array([272,  62,  54,  24, 175, 211, 403,  59,  22, 184])

This says "the first 272 rows in X are one query, then next 62 rows are another query, etc.".
Given data in this format, LGBMRanker can be used to fit a learning-to-rank model.
rnk = lgb.LGBMRanker(
    n_estimators=100,
)
rnk.fit(X=X, y=y, group=g)

To check the in-sample fit, you can use something like spearman correlation, which checks how well the ordering of predicted scores matches the actual ratings.
round(spearmanr(y, rnk.predict(X)).correlation, 5)
# 0.21626

In the Lambdarank application, LightGBM doesn't give equal weight to all positions in the ranking. For example, it will give higher preference to splits that help it choose correctly between the 1st and 2nd most relevant items than splits that help it choose correctly between the 4th and 5th most relevant movies.
This is where the label_gain parameter comes in. That parameter describes how much more importance LightGBM places on the ordering of different items.
For example, in this dataset with 5 possible ratings, something like the following...
label_gain = [1, 2, 4, 8, 16]

says "correctly ordering the first and second most relevant items is twice as important as correctly ordering the second and third most relevant items".
I encourage you to try with different values of this parameter, like this:
rnk = lgb.LGBMRanker(
    n_estimators=100,
    label_gain=[1, 2, 4, 8, 16]
)
rnk.fit(X=X, y=y, group=g)

I hope these examples help! I am going to close and lock #5297 and #5283. If you have other questions about this topic, please ask here.
If you have questions about other LightGBM topics, please open new issues and provide all the information asked for in the issue templatte.

cc @shiyu1994 @StrikerRUS @ffineis please correct me if anything I've said above is imprecise or incorrect

@sathyarr I think you are correct about label_gain=[1, 1, 1, 1, 1] represents equal importance of all items we are ranking.
About your second topic i think label_gain refers to number of ratings (label mapings), not the number of items being rated, based on Error raised in this issue: Label x is not less than the number of label mappings (y)
So if you have n ratings in your dataset you should have at least same number of values in label_gain list, irrespective  of number of items being rated. I can confirm it works when number of items is higher than number of label_gain values.
I think the misunderstanding comes from this:

says "correctly ordering the first and second most relevant items is twice as important as correctly ordering the second and third most relevant items".

and it is more correct to say: correctly classifying items between first and second highest score is twice as important as correctly classifying items between second and third highest score
This makes sense since model returns score and not direct ordering and multiple items can have same value.
I would be very grateful if @jameslamb or someone with greater understanding of model can confirm or deny this, thanks

Re-opening this since there are unanswered questions, but I personally would need to do some research before providing an answer.


Users with Team role can download images

Adds utils userIsTeam and userIsAdminOrTeam
Updates all admin checks related to images to allow both Admin and Team roles

Addresses https://artsyproduct.atlassian.net/browse/PLATFORM-2492

Logic has been updated to use roles over user type. Depends on https://github.com/artsy/force/pull/6590


Error Dropbox API error 0 from POST

Hi,
Since today I receive the following error when i try to write a file on dropbox:
Error Dropbox API error 0 from POST https://api-content.dropbox.com/1/files_put/auto/IKC/1444943671991.ikc?access_token=TOKEN :: (no response)
Here the code snippet
The initialisation of the client
"var apiClient = new Dropbox.Client({key: '0bwuev9q75n71j7'});"
And the usage
"writeFile: function (fileName, content) {
apiClient.writeFile(("IKC/" + fileName + ".ikc"), content, function (error) {
if (error) {
logMessage("Error " + error);
}
else {
alert("File written!")
}
});
}"
The parameter "filename" is a timestamp and "content" is a json as string.
Maybe someone else also had this problem and know how to fix it

At first glance, I would guess this is a networking error. Are you able to reproduce the problem?

It seems like there was a little timeout. The following code worked for me:
setTimeout(function(){
if (error) {
logMessage("Error " + error);
}
else {
alert("File written!")
}
},200);


ref(contrib): use merges for changelog summmaries

Instead of using individual commits to generate the changelog, we can use the merged PRs which better reflects the changes in a release.
refs #3509
For example, the new output:
><> ./contrib/util/generate-changelog.sh v1.5.0
### v1.5.0 -> HEAD

#### Features

 - [`fd3fc8e`](https://github.com/deis/deis/commit/fd3fc8e392e0abc361b77c51d1c5c23ebe4f0f3a) controller: add users:list endpoint
 - [`4070755`](https://github.com/deis/deis/commit/4070755d502729e3a247c80176cb2d6e15908f65) contrib/azure/azure-coreos-cluster: add an https enddpoint on c… …reation
 - [`ef3b89c`](https://github.com/deis/deis/commit/ef3b89c97e7be8fbb6c41f4f2bb4607b9ea3c2de) client: check controller before attempting to register. Fixes #3224

#### Fixes

 - [`8e960ca`](https://github.com/deis/deis/commit/8e960ca203bd5d8c48e129c4172dd2b13cdc4439) (all): use "confd --interval 5" instead of "--watch"
 - [`ce21c3b`](https://github.com/deis/deis/commit/ce21c3bfa1e71a140e9f5e259e37b1a97de5c54d) controller:  fix regex for deis limits
 - [`fa48fd2`](https://github.com/deis/deis/commit/fa48fd21e9aed44f7c50c43211b96dfa4c28f21e) controller:  remove domain
 - [`49d9002`](https://github.com/deis/deis/commit/49d9002e1cd2b1b0cac60c1aa6d3926c994fb3fb) builder: exit 1 when gitreceive captures a build
 - [`bd8b2df`](https://github.com/deis/deis/commit/bd8b2dff4b03efa66d77f22078f6d2ccceceb00f) router: use nginx $host in HTTPS redirects
 - [`def0d96`](https://github.com/deis/deis/commit/def0d9607e58cada0555a99ce86dd1e682954c8d) 'contrib/azure/azure-coreos-cluster': make this not found error more user friendly
 - [`e153834`](https://github.com/deis/deis/commit/e1538349fad1be04d906ecb04ce2f1674b3abfed) database+registry: check for S3 bucket name before creating it

#### Documentation

 - [`54757a0`](https://github.com/deis/deis/commit/54757a08e5f714789c0ead8307736849e2412136) contrib: organize community contributions
 - [`6fe16b1`](https://github.com/deis/deis/commit/6fe16b1f90b6902cd842c8d3567af60d135c7d50) installing_deis/aws: update AWS provisioning example output
 - [`b48ced3`](https://github.com/deis/deis/commit/b48ced3e3ad2c964634fcefd5f8cb56f712677a7) troubleshooting_deis: use private key on ssh -i
 - [`ce4f8c3`](https://github.com/deis/deis/commit/ce4f8c3b97e7f6c6289d80f4010096776c78bf8a) managing_deis: add upgrade Deis clients for in-place upgrade

#### Maintenance

 - [`6702716`](https://github.com/deis/deis/commit/670271697c236743dfcfd3e8d4f85f1fa6922d35) (all): bump confd to v0.9.0
 - [`82cf84c`](https://github.com/deis/deis/commit/82cf84c6c8c7bd6c898d98df41d8b325620106e2) controller: update docker-py to 1.1.0
 - [`1cdf41e`](https://github.com/deis/deis/commit/1cdf41e929d12f8f98c5ac638b1294c0243a9a9b) (all): update pip installer tool to 6.1.1
 - [`d120816`](https://github.com/deis/deis/commit/d1208162f232b1f7309b1dc4f2b98bcb76376233) contrib/ec2: introducing c4 instance types
 - [`da4408b`](https://github.com/deis/deis/commit/da4408b87eafce9c5d00e63bf0b716add1de3080) store: bump Ceph to "hammer" release
 - [`fb081f8`](https://github.com/deis/deis/commit/fb081f86cd1d9b5671bc489d71a64123d8d0fddd) client: update python-dateutil to 2.4.2
 - [`ae7adc1`](https://github.com/deis/deis/commit/ae7adc16f21a0eff5685d86627fdb8fe6c577d79) logspout: switch to busybox

I agree: this will be a more useful summary of work done on Deis. I only wonder if we should go back and regenerate the entire CHANGELOG.md, or if we should just use this going forward.
Code LGTM.

@carmstrong, I defer to thee. ^

I only wonder if we should go back and regenerate the entire CHANGELOG.md, or if we should just use this going forward.

I don't like the idea of rewriting history. I think people will understand the change. :shipit:


fix: Add Solution in Curriculum/Challenges/English/08/CodingInterviewProblems/DataStructures/CREATE_LINKED_LIST_CLASS.md

[x ] I have read freeCodeCamp's contribution guidelines.
[ x] My pull request has a descriptive title (not a vague title like Update index.md)
[ x] My pull request targets the master branch of freeCodeCamp.
[ x] None of my changes are plagiarized from another source without proper attribution.
[ x] All the files I changed are in the same world language (for example: only English changes, or only Chinese changes, etc.)
[x ] My changes do not use shortened URLs or affiliate links.

If what I am doing in the curriculum directory is ok, I will also try to fix the guide directory for this problem as well that is where the stub is and I can see what the change will look like.  I updated the guide directory previously for the AdjacencyList challenge and the pull request was merged not too long ago.  Thank you.

@thecodingaviator @RandellDawson Are fixes like this unnecessary?  Thank you.

@StanimalTheMan these foxes are fine until the solutions are alright and someone else has not already taken it up

@StanimalTheMan Can you please make the changes requested?

I have made my suggested changed on behalf of the user, since the user had a valid solution but I felt the variable names should be similar to what is used in some of the other linked list solutions.


Asus P8Z68-V-LX DSDT Patch

Hi to all,
i search to apply DSDT patch to bios Asus P8Z68-V-LX.
When compile with iasl DSDTMod.dsl, i have this error:
DSDTMod.dsl  11512:         Name (_HID, "ABCDEFGH")  // _HID: Hardware ID
Error    6035 -                                 ^ _HID suffix must be all hex digits (GH)
Can i help me.
Very Very thanks a lot.

It says at bottom of page how to fix this


⚠ ClamAV has degraded performance

In d680d43, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 221 ms

Resolved: ClamAV performance has improved in 52f3c6a.


Fixing IMG path in gulp.watch (2)

Gets also images in subfolders

During a run npm zip, subfolders don't seem to be added.

Good one!


[Request] windowsdesktop-runtime: Change release from Current to LTS

Package Request
Information
Name: windowsdesktop-runtime
Description: Looking at the release cadence from the developer Microsoft, it's clear that the Current release doesn't overlap, which means it's not only out-of-date, but also not secure to use in the future;
https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core#cadence
Problem is I haven't found a reliable way yet to make the manifest non-dependent of manual updates in regards to the Checkver URL (and the corresponding eventual regex).
https://dotnet.microsoft.com/en-us/download/dotnet/6.0 could be used to check the line ".NET Desktop Runtime < version >" with regex, but eventually that URL would have to be updated manually when v8 is released (and so forth), hence I don't find that a satisfactory solution to the issue.
The Github and its release page for the project includes every kind of version/release and is stating v5 as the latest release even now; https://github.com/dotnet/runtime
The fact that '(recommended)' is present on the Homepage could possibly be used with some form of regex since the full version number is displayed to the right, but I'm personally not sure how to do it.
Homepage: https://dotnet.microsoft.com/en-us/download/dotnet
Download link(s):
At the time of writing:
x64: https://dotnet.microsoft.com/en-us/download/dotnet/thank-you/runtime-desktop-6.0.5-windows-x64-installer
x86: https://dotnet.microsoft.com/en-us/download/dotnet/thank-you/runtime-desktop-6.0.5-windows-x86-installer
Some indication of popularity/repute:

LTS is present in the Versions bucket: https://github.com/ScoopInstaller/Versions/blob/master/bucket/windowsdesktop-runtime-lts.json
We can move it here, if required

I don't know about required, but definitely recommended based on the reasons mentioned.

Both are now in Extras


Enhance footer

Is your feature request related to a problem? Please describe.
The footer section in not properly designed, anchor tags are not styled that's why It's very tough to read what written.
I want to enhance the footer section. Kindly assign me this under gssoc23
Describe the solution you'd like

Describe alternatives you've considered
No response
Additional context
No response

You can check #26 and propose your changes there


chore: remove last legacy workspace example

Removes the last remnants of legacy workspace support.

Closes #1252.

Closes #1236.

@mergifyio queue


Hide most missed characters when there are no mistakes.

Output if there's a mistake
❯  echo "whats up" | ./sweet -
results of stdin:
wpm:                 54
uncorrected errors:  0
duration:            2.203770009s
mistakes:            1
accuracy:            90.00%
most missed keys:    s (1 time)
graph:
 60 ┼╮
 54 ┤╭╮
 48 ┤││
 42 ┤│╰
 36 ┤││
 30 ┤││
 24 ┤││
 18 ┤││
 12 ┤││
  6 ┤││
  0 ┼╯╰

     ■ raw wpm   ■ wpm

Output if there isn't a mistake
❯  echo "whats up" | ./sweet -
results of stdin:
wpm:                 93
uncorrected errors:  0
duration:            1.16496156s
mistakes:            0
accuracy:            100.00%
graph:
 96 ┼╮
 86 ┤│
 77 ┤│
 67 ┤│
 58 ┤╰
 48 ┤│
 38 ┤│
 29 ┤│
 19 ┤│
 10 ┤│
  0 ┤╰

     ■ raw wpm   ■ wpm

In short, viewing PR comments with the gh cli is a pain. re also didn't work as intended.


Add aks envs

Context
Add config files for aks environments
https://trello.com/c/jyheDesS/177-register-replicate-each-environment-to-aks
Changes proposed in this pull request

Add terraform and rails config files for each aks env
update Makefile
updates to workflows

Guidance to review
make env deploy-plan
Important business

[ ] Does this PR introduce any PII fields that need to be overwritten or deleted in db/scripts/sanitise.sql?
[ ] Does this PR change the database schema? If so, have you updated the config/analytics.yml file and considered whether you need to send 'import_entity' events?
[ ] Do we need to send any updates to DQT as part of the work in this PR?
[ ] Does this PR need an ADR?

NB: Please notify the #twd_data_insights team and ask for a review if new fields are being added to analytics.yml

 Register - Replicate each environment to AKS


Performance Producer with no message duplication and error handling

Description
Hi All ..i am making a dot.net core application which reads data from a database and sends the data to a kafka topic. (the application will be called in parallel multiple times and each app needs to send more than 2 million messages.
I need that:
All messages to me sent with high throughput.
All messages to be sent without duplication.
My questions are:
1.Should I use the internal kafka retry mechanism + a custom retry mechanism + EnableIdemPotence ?
2.What can i teak in my producer configuration to have the best throughput ?
3.In case the internal kafka retry mechanism fails, for what kind of exceptions should the application try to resend the messages so that i do not duplicate any message.
4.I have errors logged on the on producer.OnError event, like :
localhost: 7 request(s) timed out: disconnect (after 129151ms in state UP)
localhost:9092/1: 16 request(s) timed out: disconnect (after 136150ms in state UP)
In this case, or in case i have other errors logged, can i identify the messages for which the errors was triggered ?
Are the errors thrown here only information messages and i should receive for all messages sent a delivery report with the status of the message ?
5.The overall idea is that i want to identify all messages that were not sent with success to kafka so i can resent them, but without duplicating them.
Is this possible?
The kafka producer config is :
_producerConfig = new ProducerConfig {
BootstrapServers = "localhost:9092,localhost:9093,localhost:9094",
Acks = -1,
MessageSendMaxRetries = 1000,
RetryBackoffMs = 1000,
LingerMs = 1000,
EnableIdempotence = true
};
The sending logic is as follows :
//Get data from database
batch.SubscribersList = DataAccess.GetDataForId(id);
foreach (var subscriberInfo in batch.SubscribersList)
{
string kafkaKey = CreateAvroString(subscriberInfo.Id);
string kafkaValue = CreateAvroString(subscriberInfo.Value);
try
{
Retry.Execute(() => _kafka.BeginProduce(kafkaKey,kafkaValue),
CustomKafkaSendRetryIntervalMs, CustomKafkaSendMechanismRetryNo, details, _logger);
}
catch (RetryFatalException ex)
{
//log errror
}
The method used to send messages is as follows :
public void BeginProduce(string key, string value)
{
Action<DeliveryReportResult<string, string>> handler = (DeliveryReportResult<string, string> r) =>
{
if (r.Error.IsError)
{
_loggerInput.Warn($"DeliveryReport with error for {_profileID},Session:{LoadSession},Key:{r.Key},Message:{r.Message}.Error:{r.Error.Reason}.");
//add message with delivery reports error to be send again
AddMessagesWithErrors(r.Key + "-" + r.Value);
}
else
{
_totalReceivedDeliveries++;
}
};
try
{
_producerAvro.BeginProduce(_kafkaSettings.Topic, new Message<string, string> { Key = key, Value = value }, handler);
}
catch (KafkaException kex)
{
      throw kex;
}

}
In case the BeginProduce throws an kafkaexception i have a retry mechanism that sends the messages  again .
Also messages for which there is a delivery report with error (r.Error.IsError)) are sent again using a custom retry mechanism.
I Also log errors that are trigerred by the producer's OnError event.
private void _producerAvro_OnError(object sender, ErrorEvent e)
{
_loggerInput.Error($"Producer Error.Profile:{_profileID},Session:{LoadSession}.Code:{e.Code}.Reason:{e.Reason} ");
}
Checklist
Please provide the following information:

[ ] Confluent.Kafka nuget version: 1.0.0-beta2
[ ] Apache Kafka version: 0.11
[ ] Client configuration:
[ ] Operating system:Windows 10
[ ] Provide logs (with "debug" : "..." as necessary in configuration)
[ ] Provide broker log excerpts
[x ] Critical issue

Yes, the Producer's builtin retry handling should be sufficient. And enable.idempotence=true will be required to avoid duplicates (in case of retries).
The most important configuration property for performance is to increase linger.ms, throughput increases drastically up to around 100ms where it typically flattens out.
librdkafka exposes a message persistance status (rd_kafka_message_status()) that can tell you if manually retrying the message may risk duplication. I'm not sure .NET exposes it yet? (@mhowlett ). Typically you should rely on the idempotent producer to do retries for you though, so I don't think application retry logic is really needed.
The timeouts will trigger retries, so those logs are more an indication that the network or kafka has problems (keeping up). You should rely solely on the delivery report to know if a message was successfully produced or not.
The idempotent producer does this automatically, you don't need any application logic for it.

Thanks for the reply...as far a I know rd_kafka_message_status() is not exposed for .net. Is there any chance to be exposed in the near future for the .net ? @mhowlett .

i've just implemented this in a local branch. it will be in -beta3, along with correct notification of fatal errors (-beta2 always sets fatal flag to false).

https://github.com/mhowlett/confluent-kafka-dotnet/pull/9

thank you ...I do appreciate that you've added this to the .net implementation...waiting for the -beta3.
A final question.
Another thing.
Let's say that i enable idempotent, but i do not set the QueueBufferingMaxMessages ,LingerMs and BatchNumMessages for the producer.
In this case the error Local: Queue full will be generated when calling producerAvro.BeginProduce.
Is it ok to make an retry mechanism to resend the message in this case ?
Are there any other errors for which I should manually retry the producerAvro.BeginProduce without message duplication ?

yes, if you get a local queue full error, you should retry as the message is yet to get to the point where librdkafka is managing it.
better documentation of error scenarios and providing good examples / guidance on best practice is a high priority and on the roadmap for Q1 next year.

One more thing...about the producer OnError event.
Beacause -beta2 always sets fatal flag to false... is there a list of codes for which the sending mechanism
should stop .. (because the kafka server is not responding anymore without a manual intervention ) ?

excerpt from what will be in -beta3 readme should answer your question:
https://github.com/mhowlett/confluent-kafka-dotnet/blob/d97b4c3855c0b5a1feab44a2389a7f74fc009dde/README.md#error-handling

Hi ...any updates for when -beta3 will be released ?


Instantiate 128kb of scratch space in GPU memory per-device by default

For things such as the new CatArray kernel, it will be helpful to have a larger buffer of GPU scratch space available to serve as reusable GPU memory, without requiring use of the caching allocator. This diff bumps the default scratch space per device from 16 bytes to 128 kb.
cc @wickedfoo

looking forward, do we see a regime in which we dont use the caching allocator? I dont think so.
If this scratch space only helps catArray, seems wasteful. I'll accept it and merge it in, but can you rethink the reason for the scratch space.

for example, the scratch space we need for batchedGemm, we just allocate on the fly, and we assume that caching allocator is used.

I think cutorch core algorithms should not depend upon what kind of allocator is being used, and potentially suffer bad performance if the allocator is not the caching allocator.
The cutorch library might be used embedded in other systems that are managing the GPU memory on top; e.g., caffe2 or other systems. Or, the tensor library can be a drop-in C/C++ replacement for N-dimensional GPU array manipulations elsewhere.

mergefornow!


More security vendors are now tagging lasim as (password stealing) Trojan

OS: Windows 11 Pro
File: Lasim.exe
Lasim version: v0.1.2
VirusTotal analysis: https://www.virustotal.com/gui/file/9cbddd9276b5ff0fb3bcf982f766f533053c6852162b314a6536faab6bce0da5/detection
Lasim version: v0.1.1
VirusTotal analysis: https://www.virustotal.com/gui/file/3da7727225c201328391fa44eca060f6e060978dbbd7b0a163d5a083e5d81bc5/detection
FYI.  Not sure if you can do anything with it.  Not sure if it is an issue.

Yeah that's really odd. I don't know what would be tripping it up about 0.1.2, other than that it's newer.
I can definitely find references to false positives on virustotal:

https://www.reddit.com/r/rust/comments/qa7tqk/rust_executable_flagged_as_malware_by_virustotal/
https://www.reddit.com/r/golang/comments/k5ac91/tcp_chat_application_being_detected_as_virus/

So in short, 3/70 deciding it's a virus is not unheard of. Especially for a program that actively prompts users for credentials I guess.
I always use the versions compiled by GitHub, so even if my personal machine was infected somehow, what gets uploaded should be safe.
Obviously for anyone concerned, you can always examine the code and then build it from source:

https://www.rust-lang.org/tools/install
https://doc.rust-lang.org/cargo/commands/cargo-build.html

Or download the results from Github itself: https://github.com/CMahaff/lasim/actions
(i.e. the latest 0.1.2 is here: https://github.com/CMahaff/lasim/actions/runs/5549283894)
As for fixing this, not sure what else to do except look into signing the windows version, which from the above threads, might make them shut up.

Hah, just hit "re-analyze" on the same file and a completely different set of matches are now shown, 4/70.
(sorry, I think it removed your original scan results - I thought it would generate new ones).
I'm prepared to close this since the results seem kind of random, unless there is an action you'd like me to take.

OK.  I can't think of any suggestions anyway.  Thx for responding.


Soft Wrap

Would be cool if you could add soft wrapping parameter. It could append these styles to .blob-code-inner: word-wrap: normal; word-break: break-all;

Feel free to submit a patch, it looks like you just need to set white-space to normal on .gist .blob-code-inner and it will wrap.


Improve test coverage

We have to cover all the branches! 😛

Haha we look awesome from this report 👍

So with the latest master branch jest is crashing for me as if it isn't running through babel
ala
/Users/Koleok/meteor/meteor-redux-middlewares/src/utils.spec.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){import { isFSA } from 'flux-standard-action';
                                                                                ^^^^^^
    SyntaxError: Unexpected token import

Did some config get knocked out? I never bothered to set this up on circle CI so shame on me, because of that i haven't actually ran the test suite since i last worked on this repo and would have to just look back through commits to see what is up

If you're on the master branch, it's my fault: you have to remove this part of the babel configuration to run the tests. Sorry about that!
I added this because I don't want the jest files to be published on npm when I run npm run build, but I fixed the problem by putting the specific configuration inside the development env. Now we are able to run test and build without tweaking our babel configuration each time.

414e6ea

Ah forgot about this dangler!


[10.x] Added Chrono Helper

Overview
The idea is to have a fast and simple tool available to track performance issues (time or memory) on app.
As a static class, it can be imported using use Illuminate\Support\Chrono in any project file and can be used directly without any prior adjustment.
Load
The helper can be loaded with use Illuminate\Support\Chrono.
Methods and Examples
public static function reset(bool $memory = false): void

Resets timestamps and cache. Optionally resets memory peak usage. Can be used before start first measure, and when measure finish to reset the stored data.
Chrono::reset();
Chrono::reset(true); // Resets memory peak usage too.

public static function dumpKeys(string|array|null $dumpKeys): void

Sets keys to be included when dumping stamp data.
Chrono::dumpKeys(['absolute', 'relative', 'memory']);

public static function stamp(bool $dump = false, mixed $data = null): array

Creates a performance timestamp. Optionally dumps to screen the stamp point.
Chrono::stamp();

Chrono::stamp(true, ['customData' => 'value']);

public static function stamps(): array

Retrieves all stored timestamps.
$stamps = Chrono::stamps();

public static function stampsBy(string $key, int $limit = 0): array

Returns sorted timestamps based on a specified key.
$stampsByMemory = Chrono::stampsBy('memory', 5);

These examples illustrate basic usage.
Simple Full Example
<?php
use Illuminate\Support\Chrono;

function slowProcess(): void
{
    Chrono::reset();
    Chrono::dumpKeys(['absolute', 'relative', 'memory']);

    Chrono::stamp();

    foreach ($this->items() as $item) {
        OtherObjectLocal::doImportantStuff($item)
    }

    Chrono::stamp();

    foreach ($this->moreItems() as $item) {
        OtherObjectRemote::doMoreImportantStuff($item)
    }

    Chrono::stamp();

    dump(Chrono::stampsBy('relative', 5));

    Chrono::reset();
}

// OtherObjectLocal.php

use Illuminate\Support\Chrono;

public static function doImportantStuff(array $item): void
{
    Chrono::stamp();

    static::sendToLocal($item);

    Chrono::stamp();
}

// OtherObjectRemote.php

use Illuminate\Support\Chrono;

public static function doMoreImportantStuff(array $item): void
{
    Chrono::stamp();

    static::sendToRemote($item);

    Chrono::stamp();
}

Considerations
It is important to consider its use in environments like Octane, as being a direct static class, it could accumulate a large amount of data over time if the reset is not applied correctly. Since I am not fully familiar with the framework, is there any implementation that performs auto-cleaning for each individual execution/request?

How does this differ to Benchmark?

@jbrooksuk They serve different purposes. Benchmark is used to test a particular piece of code against X iterations. Chrono is used to leave marks for different parts of the code and analyze the execution time between them. You could even leave it in production and save the times in a log or send them to a remote monitoring service.

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.


Used new hashed keywords, buffered file read

18% - 25% file read speedup,
more clear code,
keywords list

Super cool! I'll review code and check the reason of build failure as soon as possible.

please wait. now I want to implement some ideas and after merge when I implement and say. You can read my sources from last commits

Now I want to refactorize line read and process - rewrite in c-style and minimize chars copy

Firstly, it uses C++11 feature(unordered_map). I want keep tinyobjloader master C++03 compilable, thus will create experimental or C++11 branch and merge the PR to there.

so I can use map :-)

map remake done!
works fast and ok

now I publish remake with hashed material_map.
from tokens or maretial.name calculate hash val and use it as key.
for red-black trees int compares if faster then string compare.
x31 hash function is good for speed and stability

Cool! > map remake done!

I have merged PR into tigrazone-master branch, and there is one error in unit test.
-------------------------------------------------------------------------------
pbr
-------------------------------------------------------------------------------
tester.cc:366
...............................................................................

tester.cc:380: FAILED:
  REQUIRE( 0.2 == Approx(materials[0].roughness) )
with expansion:
  0.2 == Approx( 0.0 )

Can you please investigate the reason of the failure, @tigrazone ?

I dont know what a problem. need to read sources. I reply you

i fix errors. try to retest.
on my computer tests passed ok

When its ready to start merge? Is there further commits planned?

two days I need to check one idea. I say you when to start merge

My ideas has no effect.
Please merge with my last commit from my repository

Merged into tigrazone branch.
It requires some amount of code refactoring, and also there is slight API change, thus I may merge this together with PR  https://github.com/syoyo/tinyobjloader/pull/151

ok i'm happy to read this!

I have merged tigrazone and https://github.com/syoyo/tinyobjloader/pull/151 into devel branch(which will become future v2.0.0). I'll do code refactoring and API cleanup from now on.


support non-collection return types

fixes #1471

Codecov Report

Merging #1480 into master will increase coverage by 0.08%.
The diff coverage is 88.23%.

@@             Coverage Diff              @@
##             master    #1480      +/-   ##
============================================
+ Coverage     69.21%   69.29%   +0.08%
- Complexity     1549     1560      +11
============================================
  Files           220      220
  Lines          6012     6025      +13
  Branches        616      621       +5
============================================
+ Hits           4161     4175      +14
+ Misses         1589     1586       -3
- Partials        262      264       +2

Flag
Coverage Δ
Complexity Δ

#integration
?
?

#unittests
69.29% <88.23%> (+0.08%)
1560 <5> (+11)
:arrow_up:

Impacted Files
Coverage Δ
Complexity Δ

...tastore/repository/query/DatastoreQueryMethod.java
41.66% <100%> (+41.66%)
4 <3> (+4)
:arrow_up:

...store/repository/query/PartTreeDatastoreQuery.java
80.14% <86.66%> (+0.14%)
45 <2> (+7)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c8faf2...a1e5688. Read the comment docs.

I think that it should be expected to work, since it is part of spring data. But maybe a couple of examples could be useful.


[BUG] Uncaught TypeError occurring causing page to not display correctly

Detailed description

Describe in detail the issue you're having.
In the Performance Env, site pages are not displaying correctly. Observed 'Uncaught TypeError' in devtools. (screen_shot_3)

Previously, was also observing a sage related error where font could not be accessed due to CORS policy (screen_shot_2). The CORS policy error has been resolved by PE but the 'Uncaught TyperError' persists.

Is this issue related to a specific component?
Sage
What did you expect to happen? What happened instead? What would
you like to see changed?
Expected result is for the site pages to load properly with no error in devtools.

What browser are you working in?
chrome

Steps to reproduce
Describe the steps to reproduce the behavior:

Navigate to: https://kajabi-performance.perf.mykajabi.farm/store (screen_shot_1)

Screenshots
Provide screenshots, videos, or gifs of issue.

updated page in editor and page loads correctly. thank you


[REQUEST Membership]: New membership for chrisliu1995

GitHub Username
chrisliu1995
Organization you are requesting membership in
openkruise
Sponsors
@FillZpp
Requirements

[x] I have joined in the community Slack or DingTalk
[x] I often attend OpenKruise Community Week
[x] I have reviewed the community-membership docs
[x] I have spoken to my sponsors ahead of this application, and they have agreed to sponsor my application
[x] I have checked my affiliation in devstats (if it's not correct, please fix in cncf/gitdm)
[x] I am actively contributing to 1 or more OpenKruise subprojects

List of your contributions

PRs reviewed / authored:

https://github.com/openkruise/kruise-game/pull/2
https://github.com/openkruise/kruise-game/pull/7
https://github.com/openkruise/kruise-game/pull/8
https://github.com/openkruise/kruise-game/pull/9
https://github.com/openkruise/kruise-game/pull/10
https://github.com/openkruise/kruise-game/pull/11
https://github.com/openkruise/kruise-game/pull/16
https://github.com/openkruise/kruise-game/pull/19

Issues :

https://github.com/openkruise/kruise-game/issues/15

+1 welcome! Thanks for building up and actively contributing to kruise-game.

+1 welcome onboard ! Thanks for contributing to kruise-game.

@chrisliu1995 Congratulations! 🎉
You should receive an email that invite you to become member of OpenKruise.
And remember to change your organization visibility to public. https://github.com/orgs/openkruise/people
/close


v1 rewrite (or v0.3.0?)

The last few months working on this bot has been a blast, though with my tendency to move fast and break things there has been quite a few growing pains which I want to address with an incremental rewrite.
This is the master issue tracking everything I want to address for the next version of the bot.
Quality of life
#24 #25
Privacy issues
https://github.com/beerpiss/chuninewbot/issues/6
Developer experience
#23 #25

Currently putting effort into a full Kotlin rewrite instead.


Safe shutdown

siad should listen for signals and handle them properly. Specifically, it should catch a ^C and initiate a graceful shutdown. This is crucial to prevent corruption of things like the wallet file.
Speaking of which, there's another step I think we should take to protect the wallet file. When the file is saved, it should be saved to a different file, like __wallet.new, and then we should call os.Rename to overwrite the old file. We can rely on the move operation to be atomic, so if siad gets interrupted it won't corrupt the old file.

Both sound good. Might actually want to make an alternate SecureSave helper function or something. We're going to have a lot of files that will be overwritten:
the host has state that needs to be saved
the hostdb has state that needs to be saved
wallet, of course
the peerlist needs to be saved
Each should go into it's own file. A general utility for save overwriting would be great to have.

The signal handling side of things is now implemented, so I'm renaming this issue.

Components should probably have a Shutdown method. Right now we have no way to, e.g., call wallet.Save because Save isn't actually part of the Wallet interface.


hmac settings failed

Describe the bug
reference the link https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/modsecurity-advanced-topic-of-the-week-hmac-token-protection/
I try to configure hmac in modsecurity, but the homepage is loaded disorderly。
Logs and dumps
Output of:

DebugLogs (level 9)
AuditLogs
Error logs
If there is a crash, the core dump file.
debug log
modsecurity_debug.log
audit log
modsec_audit.log
nginx error log
error.log

Notice: Be carefully to not leak any confidential information.
Expected behavior
A clear and concise description of what you expected to happen.
Server (please complete the following information):

ModSecurity version (and connector): 2.9.2
WebServer: [e.g. nginx-1.15.5]nginx 1.11.12
OS (and distro): [e.g. Linux, archlinux] MacOS

hmac_debug file

hmac config file

old page

destory page

I'm going to ask forgiveness from the DevGods if I am wrong here.
Does this potentially have to do with the ModSecurity dependency of the flex package and configure option "--enable-parser-generation"?

Hello @pengyejun ,
I haven't examined your posting in detail, but I did notice immediately:
ModSecurity version (and connector): 2.9.2
WebServer: [e.g. nginx-1.15.5]nginx 1.11.12

If you are you really using nginx with ModSecurity v2.9.2, please be aware that that is not recommended.
With nginx, it is recommended to use ModSecurity v3.x

Hi @martinhsv
thank you for your reply, But due to some historical reasons, I cannot quickly upgrade to v3 version now. Is there any other alternative?

Not really, I'm afraid.  Again, it's simply a not-recommended (and hence not-supported) deployment.
For use with nginx, please upgrade to ModSecurity v3.0.x.  Thanks.


Fix the mono runtime test config

Update the build to get things working with the latest mono net9 targets

@lewing https://github.com/dotnet/runtime/pull/103881 fixed the conflicting naming error, but now when running wit-bindgen with this PR changes I see this failure

Because of this I couldn't validate if the fix is enough.

test result: ok. 27 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 965.25s

@dicej
with this version I'm seeing
---- variants::run stdout ----
Error: failed to read component type file: D:\bytcodealliance\wit-bindgen\target\runtime-tests\variants\csharp-variants\VariantsWorld_component_type.o

Caused by:
    The system cannot find the file specified. (os error 2)

any thoughts?

@dicej with this version I'm seeing errors like
---- variants::run stdout ----
Error: failed to read component type file: VariantsWorld_component_type.o

Caused by:
    The system cannot find the file specified. (os error 2)

any thoughts?

haha, I see the problem now.  That is what I get for not looking again in the morning before.

@dicej with this version I'm seeing errors like
---- variants::run stdout ----
Error: failed to read component type file: VariantsWorld_component_type.o

Caused by:
    The system cannot find the file specified. (os error 2)

any thoughts?

As of https://github.com/dotnet/runtime/pull/103752, the *_component_type.o files have been replaced with *_component_type.wit files.  Perhaps you're using an older runtime?

@dicej with this version I'm seeing errors like
---- variants::run stdout ----
Error: failed to read component type file: VariantsWorld_component_type.o

Caused by:
    The system cannot find the file specified. (os error 2)

any thoughts?

As of dotnet/runtime#103752, the *_component_type.o files have been replaced with *_component_type.wit files. Perhaps you're using an older runtime?

Sadly I was rushing to get it working last night before heading out, I should have looked again this morning.  Sorry for the noise.

Ok, so it was also very confusing that the features were chained so the native aot test build was writing over the mono tests once the compilation issue was fixed.
the 'NuGet-Migrations failure is not related to these changes .
There do now appear to be a couple of failures on the mono side https://github.com/dotnet/runtime/issues/107212

Several of the issues I hit should be fixed in dotnet/runtime main, but it testing that will be challenging until a proper net10 sdk is produced.


Only use Git V2 protocol on xcode10.2 image

Older images have a new enough Git in /usr/local/bin/git, but not in /usr/bin/git (which is provided by Apple).
Some tools, particularly Homebrew, need to use the system Git in /usr/bin/git, so that one needs to be new enough to use the new protocol as well.

I'm still getting these errors on 10.1 errors with system git (not the homebrew git):
fatal: unknown value for config 'protocol.version': 2

What problem does this protocol.version': 2 solve?


Child component attribute updates break when using a transition component inside a teleport component

Version
3.1.5
Reproduction link
SFC Playground Reproduction
Steps to reproduce

Toggle the Enable Class checkbox to turn the class on and off. Note the child component content background is not turning red.
Untick the Transition? checkbox.
Repeat step 1 and note the child component content background is turning red.

What is expected?
The child component should have the .some-class class applied to it when used inside a transition in a teleport component.
What is actually happening?
The class is not being applied

This was working in v3.1.3 however I can't select v3.1.3 to reproduce a working version in the SFC playground as there appears to be an API change between v3.1.3 and v3.1.5 which breaks the demo.

This may be related to #4155

attrs is not reactive but it should be usable in the template. I think this is a bug but I'm not completely sure

without teleport also has same problem, see playground

This is a bug specific to useAttrs. For now you can simply bind to $attrs directly in the template.

@yyx990803 Thanks for sorting those out so quickly! That's great!
I have a few lingering questions regarding this though.
To give you a bit of context, the reason this has become an issue for me is because I have a composition that splits the attrs object in a higher-order component with inheritAttrs: false so that specific subsets of attributes can be redirected to different child components (ie. onBeforeEnter to a Transition and class to a div or something like that). Here's an SFC playground example of what I am trying to achieve.
Previously this worked by using a customRef. After the latest fix I can achieve the same result using an object with getters that point back to the original attrs object and a __vInternal: 1 flag to notify Vue to update. Here's a working SFC playground example
As far as I am aware it is quite a common use-case to redirect subsets of attributes from higher-order components to different parts of the template. In light of this, I am curious as to the following:

Is my approach to splitting the attrs object completely nuts and, if so, is there a better alternative that doesn't rely on using internal flags like __vInternal: 1?
In your experience is this a common use-case among other devs working on medium-large Vue codebases?
If 2 is true, would it be reasonable to request an official splitAttrs API shipped with Vue to perform these operations?


Why cant I add corner radius to Button and background view color? Please add an example with corner radius.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

I want to use Neopop library with corner radius to Button but the UI doesn't look good after adding corner raidus.

@ajithaisle we don't support buttons with corner radius yet.


Review existing JavaScript snippets (conventions)

Existing JS snippets needs some review and make sure that they follow the conventions we recently added.

Hi , @hhkaos  I want to contribute

Awesome @Priyanshu-bit !, I have assigned this one to you. Let us know if you need anything 😄
// cc: @kellyhutchins , @RalucaNicola

@hhkaos I have made the neccesary changes. Kindly have a look.

Thanks @Priyanshu-bit !! I will have a look later today 😄


Add required columnCount prop to ColumnSizer docs

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.66%. Comparing base (bd71003) to head (b450518).
Report is 34 commits behind head on master.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1535   +/-   ##
=======================================
  Coverage   94.66%   94.66%
=======================================
  Files          47       47
  Lines        1801     1801
=======================================
  Hits         1705     1705
  Misses         96       96

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Sign-up page displays view additional details when no extra fields available

When customizing the sign-up page to remove default fields the "View Additional Details" button remains even when there are no extra fields to display.
Steps to reproduce:

Enable sign-up screen on APIM
Remove claims under "View Additional Details" (eg. country, mobile, IM, etc..)
View sign-up page.

Suggested Fix:
<APIM_HOME>/repository/deployment/server/jaggeryapps/store/site/themes/wso2/templates/user/sign-up/template.jag
Line #109 - <a class="add-margin-bottom-4x sign-up-additional" title="<%= i18n.localize("View Additional Details")%>"><%= i18n.localize("View Additional Details")%>
Needs to be rendered only when fields array contains a field with a property of required = false.

Closing Since the issue has been stale for a long time.


ui - Inbox

an email/message inbox for containing correspondence based on topics (Dialob forms)

Updated design

Design v3

added user and representative (authorization) info
attachments not shown in thread overview, just indicator
download button for attachments in attachments view


HALON-517: Introduce SDSInhibited state.

Created by: nc6
This is distinct from SDSTransient, which represents a transient
failure on a disk. SDSInhibited represents that a higher level
failure has resulted in the disk being unavailable.

Closing as an obsolete


Raise alert on unmonitored system

In order to ensure that all systems on the network are being monitored, we should raise an alert when we detect a system which is not currently logging metrics
Acceptance Criteria:

[ ] Host with incorrectly configured metrics address raises alert
[x] Host with collectd stopped raises alert
[x] Host without any reporting raises alert

@sharms WDYT about those PRs?

Looks good, merged!

Anything else we need to do (other than roll this out to prod to close this card?)

Roll it out, create a new EC2 instance, verify alert is raised, and delete the instance

While testing this in staging today, I noticed that bosh not emit metrics for complication or errand VMs, so any time one of those is created, we will get false-positive alerts for those VMs.
I'm unsure how to solve this problem, as errand and compilation VM resources are ephemeral so whitelisting them isn't an option.
Anyone have any bright ideas?

@cnelson Is monitoring for false positives at the moment.

Assigning to @sharms to accept and close.