Fastlane Build Fails on Production in Expo 51 SDK, Fine in Development

Build/Submit details page URL
No response
Summary
I upgraded from SDK 49 to SDK 51 and I have been struggling to build on iOS production.
I use a dev client, and that builds fine on iOS and Android. Android builds fine in production, but just iOS doesn't.
Been trying to find solutions everywhere.
I've reset all my credentials. Removed examples of duplicate object keys etc... But, nothing is working.
I keep on getting this error:
Run script build phase '[CP-User] Generate updates resources for expo-updates' will be run during every build because it does not specify any outputs. To address this issue, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'EXUpdates' from project 'Pods')

    Run script build phase '[CP-User] Generate app.config for prebuilt Constants.manifest' will be run during every build because it does not specify any outputs. To address this issue, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'EXConstants' from project 'Pods')

    Run script build phase 'Create Symlinks to Header Folders' will be run during every build because it does not specify any outputs. To address this issue, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'BoringSSL-GRPC' from project 'Pods')

▸ ** ARCHIVE FAILED **

▸ The following build commands failed:

▸ 	Ld /Users/expo/Library/Developer/Xcode/DerivedData/Appname-Number/Build/Intermediates.noindex/ArchiveIntermediates/Appname/InstallationBuildProductsLocation/Applications/Appname.app/Appname normal (in target 'Appname' from project 'Appname')

▸ 	Archiving workspace Appname with scheme Appname

▸ (2 failures)

** ARCHIVE FAILED **
The following build commands failed:

	Ld /Users/expo/Library/Developer/Xcode/DerivedData/Appname-Number/Build/Intermediates.noindex/ArchiveIntermediates/Appname/InstallationBuildProductsLocation/Applications/Appname.app/Appname normal (in target 'Appname' from project 'Appname')

	Archiving workspace Appname with scheme Appname

(2 failures)

Exit status: 65

+---------------------------------------+

|           Build environment           |

+-------------+-------------------------+

| xcode_path  | /Applications/Xcode.app |

| gym_version | 2.225.0                 |

| sdk         | iPhoneOS18.1.sdk        |

+-------------+-------------------------+

Looks like fastlane ran into a build/archive error with your project.
It's hard to tell what's causing the error, so we wrote some guides on how to troubleshoot build and signing issues: https://docs.fastlane.tools/codesigning/getting-started/

Before submitting an issue on GitHub, please follow the guide above and make sure your project is set up correctly.

fastlane uses `xcodebuild` commands to generate your binary, you can see the the full commands printed out in yellow in the above log.

Make sure to inspect the output above, as usually you'll find more error information there

[!] Error building the application - see the log above

Error: The "Run fastlane" step failed with an unknown error. Refer to "Xcode Logs" below for additional, more detailed logs.

Please can anyone give any suggestions?
Managed or bare?
Managed
Environment
expo-env-info 1.0.5 environment info:
System:
OS: macOS 14.5
Shell: 5.9 - /bin/zsh
Binaries:
Node: 18.16.1 - /usr/local/bin/node
Yarn: 1.22.19 - /usr/local/bin/yarn
npm: 9.5.1 - /usr/local/bin/npm
SDKs:
iOS SDK:
Platforms: DriverKit 24.1, iOS 18.1, macOS 15.1, tvOS 18.1, visionOS 2.1, watchOS 11.1
IDEs:
Android Studio: 2024.2 AI-242.23339.11.2421.12550806
Xcode: 16.1/16B40 - /usr/bin/xcodebuild
npmPackages:
@expo/metro-config: ^0.18.11 => 0.18.11
expo: ~51.0.39 => 51.0.39
react: 18.2.0 => 18.2.0
react-dom: 18.2.0 => 18.2.0
react-native: 0.74.5 => 0.74.5
npmGlobalPackages:
eas-cli: 13.2.3
expo-cli: 6.3.10
Expo Workflow: bare
✔ Check package.json for common issues
✔ Validate global prerequisites versions
✔ Validate Expo Config
✔ Check compatible dependency versions for the installed Expo SDK
✔ Check for conflicting global packages in project
✔ Check for incompatible packages
✔ Verify prebuild support package versions are compatible
Didn't find any issues with the project!
Error output
No response
Reproducible demo or steps to reproduce from a blank project
run eas build --platform ios
From my expo project with expo ~51.0.39
{
  "name": "appname",
  "version": "1.0.0",
  "main": "node_modules/expo/AppEntry.js",
  "scripts": {
    "lint": "eslint .",
    "start": "expo start",
    "android": "expo start --android",
    "ios": "expo start --ios",
    "web": "expo start --web",
    "eject": "expo eject",
    "postinstall": "patch-package"
  },
  "dependencies": {
    "@expo/config-plugins": "~8.0.0",
    "@expo/metro-config": "^0.18.11",
    "@gorhom/bottom-sheet": "^5.0.5",
    "@likashefqet/react-native-image-zoom": "^2.2.0",
    "@react-native-async-storage/async-storage": "1.23.1",
    "@react-native-community/datetimepicker": "8.0.1",
    "@react-native-firebase/analytics": "^21.4.0",
    "@react-native-firebase/app": "^21.4.0",
    "@react-native-firebase/auth": "^21.4.0",
    "@react-native-firebase/firestore": "^21.4.0",
    "@react-native-firebase/functions": "^21.4.0",
    "@react-native-firebase/storage": "^21.4.0",
    "@react-navigation/material-bottom-tabs": "^6.2.29",
    "@react-navigation/native": "^6.1.18",
    "@react-navigation/native-stack": "^6.11.0",
    "@rneui/base": "^4.0.0-rc.7",
    "@rneui/themed": "^4.0.0-rc.7",
    "@sentry/react-native": "~5.24.3",
    "@turf/turf": "^6.5.0",
    "eslint-config-expo": "~7.1.2",
    "expo": "~51.0.39",
    "expo-application": "~5.9.1",
    "expo-av": "~14.0.7",
    "expo-build-properties": "~0.12.5",
    "expo-camera": "~15.0.16",
    "expo-checkbox": "~3.0.0",
    "expo-constants": "~16.0.2",
    "expo-crypto": "^13.0.2",
    "expo-dev-client": "~4.0.29",
    "expo-device": "~6.0.2",
    "expo-file-system": "~17.0.1",
    "expo-haptics": "~13.0.1",
    "expo-image": "^1.13.0",
    "expo-image-manipulator": "~12.0.5",
    "expo-image-picker": "~15.0.7",
    "expo-linking": "~6.3.1",
    "expo-location": "~17.0.1",
    "expo-mail-composer": "~13.0.1",
    "expo-media-library": "~16.0.5",
    "expo-network": "~6.0.1",
    "expo-notifications": "~0.28.19",
    "expo-splash-screen": "~0.27.7",
    "expo-store-review": "~7.0.2",
    "expo-task-manager": "~11.8.2",
    "expo-tracking-transparency": "~4.0.2",
    "expo-updates": "^0.25.27",
    "moment": "^2.29.4",
    "patch-package": "^6.5.0",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-medium-image-zoom": "^5.1.8",
    "react-native": "0.74.5",
    "react-native-confetti-cannon": "^1.5.2",
    "react-native-device-info": "^10.8.0",
    "react-native-dropdown-picker": "^5.4.6",
    "react-native-gesture-handler": "~2.16.1",
    "react-native-google-mobile-ads": "^14.3.0",
    "react-native-iap": "^12.11.0",
    "react-native-image-modal": "^3.0.12",
    "react-native-image-viewing": "^0.2.2",
    "react-native-keyboard-aware-scroll-view": "^0.9.5",
    "react-native-maps": "1.14.0",
    "react-native-onboarding-swiper": "^1.3.0",
    "react-native-paper": "^5.12.5",
    "react-native-reanimated": "^3.10.1",
    "react-native-reanimated-carousel": "^3.5.1",
    "react-native-safe-area-context": "4.10.5",
    "react-native-screens": "3.31.1",
    "react-native-stars": "^1.2.2",
    "react-native-vector-icons": "^10.2.0",
    "react-native-webview": "13.8.6",
    "react-native-zoomable-view": "^0.3.1",
    "react-query": "^3.39.3",
    "shorthash": "^0.0.2",
    "styled-components": "^6.0.4",
    "zustand": "^4.3.9"
  },
  "devDependencies": {
    "@babel/core": "^7.24.0",
    "@eslint/js": "^9.14.0",
    "eslint": "^8.57.1",
    "eslint-plugin-react": "^7.37.2",
    "globals": "^15.12.0"
  },
  "resolutions": {
    "@expo/config-plugins": "~8.0.0",
    "@expo/prebuild-config": "~7.0.0",
    "expo-modules-autolinking": "~1.11.0"
  },
  "private": true,
  "dev": "APP_VARIANT=development expo start"
}

1

1

Are you having the same?

Hey,
Can you send me the URL of your build?

@szdziedzic

Hey,
Can you send me the URL of your build?

I sent you a message on Linkedin with the URL.
Thanks so much!

I see this error in the expo doctor check of your project: Multiple lock files detected (yarn.lock, package-lock.json). This may result in unexpected behavior in CI environments, such as EAS Build, which infer the package manager from the lock file.
Can you try only to use one lock file?

@szdziedzic

I see this error in the expo doctor check of your project: Multiple lock files detected (yarn.lock, package-lock.json). This may result in unexpected behavior in CI environments, such as EAS Build, which infer the package manager from the lock file.
Can you try only to use one lock file?

I updated it, and it's not showing that error, but it's still failing:
Running "expo doctor"
Running 14 checks on your project...
✔ Check package.json for common issues
✖ Check Expo config for common issues
✔ Check native tooling versions
✔ Check if the project meets version requirements for submission to app stores
✔ Check for common project setup issues
✔ Check dependencies for packages that should not be installed directly
✔ Check for app config fields that may not be synced in a non-CNG project
✔ Check for issues with Metro config
✔ Check npm/ yarn versions
✔ Check Expo config (app.json/ app.config.js) schema
✔ Check that packages match versions required by installed Expo SDK
✔ Check for legacy global CLI installed locally
✔ Check that native modules do not use incompatible support packages
✔ Check that native modules use compatible support package versions for installed Expo SDK
Detailed check results:
You have an app.json file in your project, but your app.config.js is not using the values from it. Remove the static app.json, or use its values in your dynamic app.config.js. Learn more: https://docs.expo.dev/workflow/configuration
One or more checks failed, indicating possible issues with the project.

What is Sentry?

Sentry was causing the issue. I fixed the Auth token and it worked.

Can you tell me how you solved this problem? i met the same question: Build failed: The "Run fastlane" step failed with an unknown error. Refer to "Xcode Logs" below for additional, more detailed logs.

@Jessie3 it's usually best to inspect Xcode logs and look for error: logs there to find more info.


Managing Atomic Data on Git

@hoijui described a workflow that he'd like to have, where he can modify some Atomic Data (in this case an ontology of Properties and Classes), and have it hosted somewhere + updated when he commits. Seems achievable, but how?
Have a script that posts commits to an atomic server

Create a github action that reads some (predetermined) .jsonad file in a github repo and sends Commits to some predetermined server
Pass a secret (agent + private key) as an env variable
Run the Github Action whenever wished (e.g. on commit on master)

Use github pages in some clever way
Not sure how and if this is possible. It's not possible to set HTTP headers on github pages, so getting an application/ad+json header is not possible.
Use the raw files?
Github's raw API allows plain JSON objects to be hosted. We could use these for the publishing protocol #93. Downside of this approach is that we'd probably not have @ids, i.e. URLs that resolve.
Hmm.. Maybe there is a better way?

My approach for this would be to decouple git from Github(/Gitlab/whatever):

teach Atomizer to parse a git repo
run Atomizer as a git hook
if it really need to be Github-specific then have a CI job or some other service hook run Atomizer server-side at each commit

@jonassmedegaard Agreed, that's a more sensible approach.


fix(working-with-json-fields): Restructure

Split advanced and simple filtering (because only "advanced" used to be a preview feature and is also more complex)
Restructure introduction of advanced section (Actual info first, then warning, and finally version info)

Let's get the Preview deployment working.

Can you help with that @ruheni? To me the content looks fine and the error messages are in no way helpful.

Sure! I'll pick it up sometime this week.

Still not building on Vercel 😢

I spent a little more time than I should trying to figure it out. It doesn't work locally either. I'll give it another stab.

@janpio shall we take another crack at this? 🙂

Yes, I think that would still be valuable. I assume the intention behind this change is still possible with the new docs, so please take over.


fix db migrations problem

Hi,
This is a fix for a db config issue. looks like being caused by some code refactoring.
Maybe using db.knex.client.dialect will be better.

Hi there @king6cong!

This is a fix for a db config issue

Can you please take some time to describe the problem including full reproduction steps (if you hit "new issue" there's a bug report template). Without any details it's impossible to verify whether your code fixes the issue or know why this should be merged :wink:

Steps to Reproduce
when db migration is triggered, like changing the db version
--- a/core/server/data/schema/default-settings.json
+++ b/core/server/data/schema/default-settings.json
@@ -1,7 +1,7 @@
 {
     "core": {
         "databaseVersion": {
-            "defaultValue": "004"
+            "defaultValue": "005"
         },
         "dbHash": {
             "defaultValue": null

this error will be produced

Migrations: Database upgrade required from version 004 to 005
Migrations: Creating database backup

ERROR: Cannot read property 'client' of undefined

 TypeError: Cannot read property 'client' of undefined
    at Object.getTables (/home/vagrant/repos/Ghost/core/server/data/schema/commands.js:82:39)
    at exporter (/home/vagrant/repos/Ghost/core/server/data/export/index.js:31:67)
    at backupDatabase (/home/vagrant/repos/Ghost/core/server/data/migration/index.js:46:12)
    at Object.migrateUp (/home/vagrant/repos/Ghost/core/server/data/migration/index.js:164:12)
    at Runner_SQLite3.<anonymous>

I hope this helps :smile:

I hope this helps :smile:

Sorry but nope! I'm actively working on our migrations system at the moment and have been testing it quite a lot. I've not seen this error. What were you doing when this happened? What is your db config?

Sorry, my fault.
After a rerun of npm install, everything is ok.
The version of knex I used was old.
-    "knex": "0.7.3",
+    "knex": "0.9.0",

Thanks for your time :kissing_cat:


Ejected dependencies should be installed with exact versions

Description
Ejected dependencies should be installed with exact versions.
Implementation

[  ] Include -E flag during install (both yarn and npm)

@shvagen Curses!


Allow to create a new instance on the client side (browser)

I have an app that has a Job model. On the server side I can easily do new Job({worker: 1}) and get a job instance which already contains the default structure defined in the Job schema.
Now I need the same thing on the client side (a browser). I don't want to talk to the server to create a new Job instance. As is offline it should allow me to do this without even touching once the server side.
Any ideas?

Sorry for not replying before but I didn't had time to test it out. I will reply something when I will need this again.


可配置爬虫，启动报错  error create file error: open /app/spiders/baidu/md5.txt

Bug 描述
用百度那个模板，创建了一个可配置爬虫，什么都没改，启动时候报如下错误
crawlab-worker | 2020/06/04 02:14:00  info start sync spider to local, total: 4 crawlab-worker | 2020/06/04 02:15:00  info start sync spider to local, total: 3 crawlab-worker | 2020/06/04 02:15:20 error create file error: open /app/spiders/baidu/md5.txt: no such file or directory, file_name: /app/spiders/baidu/md5.txt crawlab-worker | goroutine 1609 [running]: crawlab-worker | runtime/debug.Stack(0x0, 0x0, 0x0) crawlab-worker | 	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d crawlab-worker | runtime/debug.PrintStack() crawlab-worker | 	/usr/local/go/src/runtime/debug/stack.go:16 +0x22 crawlab-worker | crawlab/utils.OpenFile(0xc000623380, 0x1a, 0xc00070f628) crawlab-worker | 	/go/src/app/utils/file.go:51 +0x145 crawlab-worker | crawlab/utils.ReadFileOneLine(0xc000623380, 0x1a, 0x0, 0x0) crawlab-worker | 	/go/src/app/utils/file.go:27 +0x5b crawlab-worker | crawlab/utils.GetSpiderMd5Str(0xc000623380, 0x1a, 0x2, 0xc000623380) crawlab-worker | 	/go/src/app/utils/file.go:39 +0x39 crawlab-worker | crawlab/services.SpiderFileCheck(0xc000038510, 0x24, 0xc000451310, 0xc, 0xbfae180e00990073, 0x23404e7029, 0x18dd4e0, 0x0, 0x0, 0x0, ...) crawlab-worker | 	/go/src/app/services/task.go:695 +0x412 crawlab-worker | crawlab/services.ExecuteTask(0x1) crawlab-worker | 	/go/src/app/services/task.go:577 +0xce1 crawlab-worker | crawlab/services.GetExecuteTaskFunc.func1() crawlab-worker | 	/go/src/app/services/task.go:429 +0x2a crawlab-worker | crawlab/lib/cron.FuncJob.Run(0xc00046e280) crawlab-worker | 	/go/src/app/lib/cron/cron.go:131 +0x25 crawlab-worker | crawlab/lib/cron.(*Cron).startJob.func1(0xc0004b20a0, 0x117a7c0, 0xc00046e280) crawlab-worker | 	/go/src/app/lib/cron/cron.go:307 +0x5f crawlab-worker | created by crawlab/lib/cron.(*Cron).startJob crawlab-worker | 	/go/src/app/lib/cron/cron.go:305 +0x73 crawlab-worker | 2020/06/04 02:15:20 error read file error: invalid argument crawlab-worker | 2020/06/04 02:15:20  info [Worker 1] start task (id:568bde1e-4e87-40d0-a257-ca2d0003f028) crawlab-worker | 2020/06/04 02:15:20  info cwd: /app/spiders/baidu crawlab-worker | 2020/06/04 02:15:20  info cmd: scrapy crawl config_spider crawlab-worker | 2020/06/04 02:15:20 error wait process finish error: exit status 1 crawlab-worker | goroutine 1609 [running]: crawlab-worker | runtime/debug.Stack(0x0, 0x0, 0x0) crawlab-worker | 	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d crawlab-worker | runtime/debug.PrintStack() crawlab-worker | 	/usr/local/go/src/runtime/debug/stack.go:16 +0x22 crawlab-worker | crawlab/services.WaitTaskProcess(0xc0001162c0, 0xc000038510, 0x24, 0xc000451310, 0xc, 0xbfae180e00990073, 0x23404e7029, 0x18dd4e0, 0x0, 0x0, ...) crawlab-worker | 	/go/src/app/services/task.go:311 +0x15c crawlab-worker | crawlab/services.ExecuteShellCmd(0xff009b, 0x1a, 0xc000646e20, 0x12, 0xc000038510, 0x24, 0xc000451310, 0xc, 0xbfae180e00990073, 0x23404e7029, ...) crawlab-worker | 	/go/src/app/services/task.go:390 +0xb25 crawlab-worker | crawlab/services.ExecuteTask(0x1) crawlab-worker | 	/go/src/app/services/task.go:611 +0x11ec crawlab-worker | crawlab/services.GetExecuteTaskFunc.func1() crawlab-worker | 	/go/src/app/services/task.go:429 +0x2a crawlab-worker | crawlab/lib/cron.FuncJob.Run(0xc00046e280) crawlab-worker | 	/go/src/app/lib/cron/cron.go:131 +0x25 crawlab-worker | crawlab/lib/cron.(*Cron).startJob.func1(0xc0004b20a0, 0x117a7c0, 0xc00046e280) crawlab-worker | 	/go/src/app/lib/cron/cron.go:307 +0x5f crawlab-worker | created by crawlab/lib/cron.(*Cron).startJob crawlab-worker | 	/go/src/app/lib/cron/cron.go:305 +0x73 crawlab-worker | 2020/06/04 02:15:20 error exit error, exit code: 1 crawlab-worker | 2020/06/04 02:15:20 error [Worker 1] exit status 1

似乎是工作节点同步出了问题

如何配置的？

删除又重建了下，问题没有了。


Unable to use an exact version

I can't seem to use an exact version of the package.
Is this normal?

When fixing version with 1.1.0, Xcode says : "Failed to resolve dependencies Dependencies could not be resolved because package 'markdownview' is required using a stable-version but 'markdownview' depends on an unstable-version package 'swift-markdown' and root depends on 'markdownview' 1.1.0."

I'm sorry. It's normal upon till now.
This framework depends on Apple's swift-markdown framework, and maybe that framework is still in progress. I can't specify a constant tag because there might have some bugs inside the package.
To solve that, you just need to switch to my main branch to get the latest update. And don't worry, I will only merge stable version of MarkdownView to main branch.

If you use Xcode to add this package, choose Dependency Rule to Branch and main.

@louis49 I've seen your PR. I'll check this out. Thanks.

Thank you a lot. Can you create a new release with this fix please ?

OK. I'll do that soon.

@louis49 Published and @MohamedXi You can try it again. Enjoy it.

@louis49 Seems like it's still not fixed because we use 'branch' dependency and there might have future pushes. (It's not a static release or static tag).

I reopened this issue because it's not fixed and maybe this won't and can't be fixed in recent.

You can get the same result by letting "main" in "main" branch and "release/5.8" in a tagged version ...

@louis49 As you mentioned, same result.

When fixing version with 1.1.0, Xcode says : "Failed to resolve dependencies Dependencies could not be resolved because package 'markdownview' is required using a stable-version but 'markdownview' depends on an unstable-version package 'swift-markdown' and root depends on 'markdownview' 1.1.0."

This issue remains. It doesn’t make sense. The only solution here is directly use main branch

I think you can do :

Merge my PR
Create a new release with a tag with that code
Roll-back the dependency version of swift-markdown.git to "main"
Push this to your main branch

=> Your main branch will work with all new fixes of swift markdown : you are happy !
=> 1.2.0 tag will keep current code fixed : I am happy !
;-)

@louis49 I understand your requirement, but I’m wondering why you need that fix? I tested, still says “failed to resolve …”

The problem is if you use a branch dependency in the Package.swift file. There is no meaning publishing a version for it since SwiftPM will reject to use it.
A fixed commit revision dependency to swift-markdown may help solve the problem.

The problem is if you use a branch dependency in the Package.swift file. There is no meaning publishing a version for it since SwiftPM will reject to use it.
A fixed commit revision dependency to swift-markdown may help solve the problem.

Yes. I must switch to a fixed commit, but #6 is trying to use a branch, which results in the same error.

swift-markdown has a official release now. I have updated my package. Fixed now. @MohamedXi @louis49 Check it out.

Thank you


Add namespace_mapping_controller

In order to replicate the init.sh script as a secret into the namespaces the matchSelector of the codeModules indicate we implemented a namespace_mapping_controller which controls a ConfigMap in which we have a direct mapping between CustomResources and namespaces to monitor.
The namespace_controller now just takes care of updating this ConfigMap and adding corresponding entries to it instead of replicating the secret. This logic will be implemented and merged into this branch by @aorcholski
Additionally a timestamp within the OneAgent status got removed.

closed because of https://github.com/Dynatrace/dynatrace-operator/pull/241


bundle: Use latest upstream image for ODF Console.

Use official images for ODF Console from quay repository.
Update CSV annotations to identify ODF Operator as a dynamic plugin carrier.

/retest

/override ci/prow/odf-operator-bundle-e2e-aws

/override ci/prow/odf-operator-bundle-e2e-aws


Does this work with RHEL/CentOS?

Curious if this can be installed/build on RHEL 5/6 or other RPM based systems.

There is no reason, why it wouldn't work, however, I didn't make any attempt to try it. Please give me some time so that I can install an eval RHEL version VM, try it (and possibly build RPM).

Ok cool. FYI CentOS is the same as RHEL and free... I've updated subject.

Resources:

https://access.redhat.com/articles/142303
https://fedoraproject.org/wiki/How_to_create_an_RPM_package
http://www.g-loaded.eu/2006/04/05/how-to-build-rpm-packages-on-fedora/

I'm on Fedora and it works just fine. Compiled from source.

Good to know. Thanks!

See https://copr.fedoraproject.org/coprs/tbabej/hstr/ and instructions in the Installation.md. I suggest this resolves the issue.

Yes, it does - thank you @tbabej!


Support timeout when fetching metadata

Versions
Sarama Version: v1.22.0-6-g2a49b70 (2a49b70a5c87e2dac5801f72d80fd2732c1c7b75)
Kafka Version: 1.0.2
Go Version: 1.12
Configuration
What configuration values are you using for Sarama and Kafka?
Default configuration, see unit tests.
Problem Description
Fetching metadata from an unreachable cluster using the default configuration (30s DialTimeout, 3 retries, 250ms backoff) of 2 brokers can take more than 4 minutes  before failing with ErrOutOfBrokers.
The following logs can be seen when running the provided unit test against master with unresponsive seed brokers (with TCP read timeout on <IP_ADDRESS>:63168 and <IP_ADDRESS>:63169 to simulate unreachable cluster with TCP dial timeout):
DEBUG=true go test -v -run=TestClientMetadataTimeout/timeout=10m0s
[sarama] 2019/04/18 09:38:00 Using random seed:<PHONE_NUMBER>041448000
=== RUN   TestClientMetadataTimeout
=== RUN   TestClientMetadataTimeout/timeout=10m0s
[sarama] 2019/04/18 09:38:05 Initializing new client
[sarama] 2019/04/18 09:38:05 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63166
[sarama] 2019/04/18 09:38:05 Connected to broker at <IP_ADDRESS>:63166 (unregistered)
[sarama] 2019/04/18 09:38:05 Successfully initialized new client
[sarama] 2019/04/18 09:38:05 Closed connection to broker <IP_ADDRESS>:63166
[sarama] 2019/04/18 09:38:05 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:38:05 Connected to broker at <IP_ADDRESS>:63168 (unregistered)
[sarama] 2019/04/18 09:38:35 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63170-><IP_ADDRESS>:63168: i/o timeout
[sarama] 2019/04/18 09:38:35 Closed connection to broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:38:35 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:38:35 Connected to broker at <IP_ADDRESS>:63169 (unregistered)
[sarama] 2019/04/18 09:39:05 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63184-><IP_ADDRESS>:63169: i/o timeout
[sarama] 2019/04/18 09:39:05 Closed connection to broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:39:05 client/metadata no available broker to send metadata request to
[sarama] 2019/04/18 09:39:05 client/brokers resurrecting 2 dead seed brokers
[sarama] 2019/04/18 09:39:05 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2019/04/18 09:39:05 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:39:05 Connected to broker at <IP_ADDRESS>:63168 (unregistered)
[sarama] 2019/04/18 09:39:35 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63187-><IP_ADDRESS>:63168: i/o timeout
[sarama] 2019/04/18 09:39:35 Closed connection to broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:39:35 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:40:05 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63193-><IP_ADDRESS>:63169: i/o timeout
[sarama] 2019/04/18 09:40:05 Closed connection to broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:40:05 client/metadata no available broker to send metadata request to
[sarama] 2019/04/18 09:40:05 client/brokers resurrecting 2 dead seed brokers
[sarama] 2019/04/18 09:40:05 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama] 2019/04/18 09:40:05 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:40:05 Connected to broker at <IP_ADDRESS>:63168 (unregistered)
[sarama] 2019/04/18 09:40:35 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63196-><IP_ADDRESS>:63168: i/o timeout
[sarama] 2019/04/18 09:40:35 Closed connection to broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:40:35 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:40:35 Connected to broker at <IP_ADDRESS>:63169 (unregistered)
[sarama] 2019/04/18 09:41:05 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63204-><IP_ADDRESS>:63169: i/o timeout
[sarama] 2019/04/18 09:41:05 Closed connection to broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:41:05 client/metadata no available broker to send metadata request to
[sarama] 2019/04/18 09:41:05 client/brokers resurrecting 2 dead seed brokers
[sarama] 2019/04/18 09:41:05 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama] 2019/04/18 09:41:05 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:41:35 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63211-><IP_ADDRESS>:63168: i/o timeout
[sarama] 2019/04/18 09:41:35 Closed connection to broker <IP_ADDRESS>:63168
[sarama] 2019/04/18 09:41:35 client/metadata fetching metadata for all topics from broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:42:05 client/metadata got error from broker -1 while fetching metadata: read tcp <IP_ADDRESS>:63230-><IP_ADDRESS>:63169: i/o timeout
[sarama] 2019/04/18 09:42:05 Closed connection to broker <IP_ADDRESS>:63169
[sarama] 2019/04/18 09:42:05 client/metadata no available broker to send metadata request to
[sarama] 2019/04/18 09:42:05 client/brokers resurrecting 2 dead seed brokers
[sarama] 2019/04/18 09:42:05 Closing Client
    --- PASS: TestClientMetadataTimeout/timeout=10m0s (240.80s)
        client_test.go:670: Got err: kafka: client has run out of available brokers to talk to (Is your cluster reachable?) after waiting for: 4m0.792878004s

This time grows exponentially with the number of brokers because of the following formula:
Net.[Dial|Read]Timeout * BrokerCount * (Metadata.Retry.Max + 1) + Metadata.Retry.Backoff * Metadata.Retry.Max

Solution
The proposed solution is to add a new configuration option Metadata.Timeout to fail faster.
Changes

add Metadata.Timeout configuration option to fail faster (disabled by default for backward compatibility)
stop retry to query metadata on next broker when reaching such timeout has expired and return error
add unit tests to validate behaviour when such configuration option is set to non 0

Testing done
See provided unit tests to see ErrOutOfBrokers returned much faster.
We have been using that approach in production successfully (based on a different Sarama release) to failover a producer from one Kafka cluster to another.
When one cluster become unreachable over WAN we can failover to another region in less than 30 seconds using the following configuration:
config.Net.DialTimeout = 10 * time.Second
config.Net.ReadTimeout = 15 * time.Second
config.Net.WriteTimeout = 10 * time.Second
// Use a single retry for querying metadata rather than 3
config.Metadata.Retry.Max = 1
// Same for produce request
config.Producer.Retry.Max = 1
// Try to connect to at most two unreachable brokers
config.Metadata.Timeout = 15 * time.Second
// Disable producer retries
saramaConfig.Producer.Retry.Max = 0
// Also disable backoff as we could sleep for 100ms for every queued message
// that need to be retried if a cluster is not reachable...
saramaConfig.Producer.Retry.Backoff = 0

I just signed the CLA.

@slaunay code looks good and straight forward. Just want to make sure that this is a desirable feature to add. @bai can you please add some folks to review who might have opinions on this?

This is nice (ale missing for me) feature, so +1

Thanks for the review, failing fast (time bound) vs tolerating faults (e.g. try all brokers) is often tricky to get right.


Add missing tests for indexes

I found that there was not enough exception testing for the names property.

So I added tests for those exceptions,
And also added tests for some other code that isn't being tested yet.
It's very appreciate it if someone check it out when available. :)
Thanks.

Codecov Report

Merging #692 into master will decrease coverage by 2.34%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master    #692      +/-   ##
=========================================
- Coverage   93.54%   91.2%   -2.35%
=========================================
  Files          32      32
  Lines        5455    5455
=========================================
- Hits         5103    4975     -128
- Misses        352     480     +128

Impacted Files
Coverage Δ

databricks/koalas/usage_logging/__init__.py
23.14% <0%> (-74.08%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
50% <0%> (-50%)
:arrow_down:

databricks/koalas/__init__.py
76.92% <0%> (-7.7%)
:arrow_down:

databricks/conftest.py
90.69% <0%> (-6.98%)
:arrow_down:

databricks/koalas/frame.py
92.95% <0%> (-1.79%)
:arrow_down:

databricks/koalas/namespace.py
87.75% <0%> (-1.64%)
:arrow_down:

databricks/koalas/plot.py
93.26% <0%> (-1.35%)
:arrow_down:

databricks/koalas/utils.py
97.31% <0%> (-0.68%)
:arrow_down:

databricks/koalas/testing/utils.py
77.35% <0%> (-0.63%)
:arrow_down:

databricks/koalas/internal.py
95.67% <0%> (-0.49%)
:arrow_down:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1b8bfb...c63c629. Read the comment docs.


Private val fix

This is the fix for https://github.com/Microsoft/visualfsharp/issues/27. Testing needs to be added.

Testing added, I believe this fix is now ready.


Use npm to install a specific npm version

Hey all,
I recently downloaded the nodejs module and I think it's really well done. I have run into a small feature request though. I'm using Ubuntu and the NodeSource packages, which (I believe) puts the npm binary in the same package as nodejs, instead of having 2 separate packages like the base Ubuntu packages.
I would like to be able to use npm to upgrade npm to a specific version. When I set $npm_package_ensure to be a specific version it errors out though because it's trying to install that version via the default provider instead of via the npm provider.
I'd love to add another variable that would allow you to override the provider for npm (something like $npm_provider).  Are there any downsides to doing this?  Is there a better way to get a specific version of npm using npm?  I'd love any feedback that the community would like to share.
Thanx!

Can you try:
nodejs::npm { 'npm':
   ensure   => 'version',
   provider => npm,
}

Looks like that errors out: Error 400 on SERVER: Invalid parameter provider on Nodejs::Npm[npm]
Looking at the nodejs::npm defined type shows that there isn't a provider parameter.
I tried this also:
  nodejs::npm { 'npm':
    ensure => '2.13.4',
    target => '/usr/bin',
  }

But that also dies, trying to install npm from the default package manager, instead of from npm.

Oh right maybe:
package { 'npm':
  ensure => '2.13.4',
  provider => npm,
}

That's what I tried at first, but I got a duplicate resource error because Package[npm] is already defined in install.pp on line 49. Now that I look at it again, maybe I can set $nodejs::npm_package_name to undef and then that whole package section won't be in affect.

Hmm, I tried this, but I still got Duplicate Declaration: Package[npm] is already declared
class profile::appserver::nodejstools {
  class { 'nodejs':
    # Other nodejs values are set in hiera, but undef can't be set in hiera
    npm_package_name => undef,
  }

  package { 'npm':
    ensure   => '2.13.4',
    provider => 'npm',
  }
}

That makes sense. This is a legitimate deficiency in the software, and one that affects me as well.

This would only be achievable in Puppet > 4 :cry: (
See https://tickets.puppetlabs.com/browse/PUP-1073 for the full story)
In Puppet > 4:
class profile::appserver::nodejstools {
  include nodejs

  package { "npm-installed-with-npm":
    name     => "npm",
    provider => "npm",
    ensure   => "2.13.4",
    require   => Class['nodejs'],
  }

}

any updates on this?
My current workaround:
    exec { 'upgrade npm to v3:
        command => 'sudo env "PATH=$PATH" npm install npm -g',
        onlyif  => "test `npm -v | awk -F \. {'print $1'}` -lt 3",
    }


Profile and Auth alert / message

Change the outlook in user profile

忘記密碼 update 完之後 Update successfully 字太小了 而且在最下面按鈕旁邊根本就看不到 -> use message
讓 user 跟 subjudge 可以看到自己的 user level 但是不能改 -> add readonly form
改user level ＋alert -> add onsubmit check

judge can update other account to admin OAQ

Please hide user email from other users except for admin.
We'll add a checking box to decide whether an user's will public or not (maybe in next release).

+1, @bruce3557 please check

+1 besides the comments

There is no other comment. Going to merge.


Beginner help wanted to setup Atom and more

Hi,
I am not sure wether I should write here, but I have to try !
I've discovered ev3 python very recently. I was looking for some way to write code instead of using the classic Mindstorms GUI app.
I must say first that I am a total newbie, but I know a few things.
I've already installed the ev3dev on my robot, and connected it via Bluetooth, I have even been able to shut it down with Terminal (I'm on a mac). Cool.
I am currently trying to learn python with https://learnpythonthehardway.org/ and the author recommends Atom to write code instead of PyCharm or any other IDLE/IDE.
I've read the github discussions about the doc not being very clear for beginners. I've also read some other things on the web, and I can't find the way to get Atom (atom.io) aware of all the ev3python library and classes (sorry if I say stupid things, I still have quite a lot to learn !)
If I try and sum things up :
ev3dev is installed and runs, my robot boots correctly, I can SSH it, and I need help to figure out how to be able to write code in an editor with the classes. Does it make sense ?
Thanks for any help, and a big thank you for giving us the opportunity to get a little cleverer (hopefully)
Romain

Did you have a look at https://ev3python.org?

Hi,
Thanks for the reply. Yes, I have read it. The thing is, I am a bit confused, because I've read so many things lately, I've tried many, and sometimes I go through some web page explaining this or that, and I don't really understand.
What I currently can't figure out is how to install/use this ev3dev-lang-python. Do I even need it ? You must be smiling reading this !

It should be already pre-installed of you flashed with a recent image. To check this, ssh to your ev3, and do this in terminal:
python3

This should open an interactive python session. There enter
from ev3dev.ev3 import *
Sound.speak('hello')

Press ctrl-d on an empty line to exit.

I would recommend reading this page as a next step: https://sites.google.com/site/ev3python/putty

Yes, it works. I've found this
https://www.gitbook.com/read/book/7hal32/ev3-tud-main/chapters/ev3dev-TUD/setup.html
but
it's finally a 404 error. The text below that google search was interesting
:
By default there will only be auto-completion for standard Python packages.
To enable it you will have to install the ev3dev languages bindings for
Python. +.
I've been able to see this website, and I've seen some interesting code on
it in order to be able to do what I'd like to, but it was at school, and
now, for some reason, I can't access it anymore.
As you have already guessed, it's a bit messy in my head, I can't even ask
the right question ! But I will finally figure it out...
Romain Schmid
Chemin Henri-Giroud 7
1446 Baulmes
079/249 24 48

Aaaaaah, finally !
I had to update pip, install Pillow, and I finally managed to get the code properly highlighted and all classes included in PyCharm.
I'd say you can close after a nice little beer. Thanks for your patience.

:beers:


replace with request with got?

considering request library is now deprecated and got is the one recommended

We're slowly migrating to @apify/http-request which uses Got underneath.


fix(typescript): Move typescript definition to the root of the project

https://www.typescriptlang.org/docs/handbook/declaration-files/publishing.html#including-declarations-in-your-npm-package
bootstrap-styled
Thank you for contributing! Please take a moment to review our contributing guidelines
to make the process easy and effective for everyone involved.
Please open an issue before embarking on any significant pull request, especially those that
add a new library or change existing tests, otherwise you risk spending a lot of time working
on something that might not end up being merged into the project.
Before opening a pull request, please ensure:

[ ] You have followed our contributing guidelines
[ ] Double-check your branch is based on dev and targets dev
[ ] Your are doing semantic commit message using commitizen and our commit message convention
[ ] Pull request has tests (we are going for 100% coverage!)
[ ] Code is well-commented, linted and follows project conventions
[ ] Documentation is updated (if necessary)
[ ] Description explains the issue/use-case resolved and auto-closes related issues

Be kind to code reviewers, please try to keep pull requests as small and focused as possible :)
IMPORTANT: By submitting a patch, you agree to allow the project
owners to license your work under the terms of the  License.

:tada: This PR is included in version 2.5.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


CMake fails when called from within VSCode

Brief Issue Summary
I have a project where I manually download googletests and build with cmake. When trying to configure the project from within VSCode it fails while detecting pthreads. Configuring manually through the terminal it succeeds.
Any idea how to fix this?
CMake Tools Log
Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /home/niko/git/handle_allocator/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_1abb2/fast && /usr/bin/make  -f CMakeFiles/cmTC_1abb2.dir/build.make CMakeFiles/cmTC_1abb2.dir/build
make[1]: Entering directory '/home/niko/git/handle_allocator/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_1abb2.dir/src.c.o
/usr/bin/gcc-7 -DCMAKE_HAVE_LIBC_PTHREAD   -o CMakeFiles/cmTC_1abb2.dir/src.c.o -c /home/niko/git/handle_allocator/build/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_1abb2
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_1abb2.dir/link.txt --verbose=1
/usr/bin/gcc-7 CMakeFiles/cmTC_1abb2.dir/src.c.o -o cmTC_1abb2
CMakeFiles/cmTC_1abb2.dir/src.c.o: In function `main':
src.c:(.text+0x3e): undefined reference to `pthread_create'
src.c:(.text+0x4a): undefined reference to `pthread_detach'
src.c:(.text+0x56): undefined reference to `pthread_cancel'
src.c:(.text+0x67): undefined reference to `pthread_join'
src.c:(.text+0x7b): undefined reference to `pthread_atfork'
collect2: error: ld returned 1 exit status
CMakeFiles/cmTC_1abb2.dir/build.make:105: recipe for target 'cmTC_1abb2' failed
make[1]: *** [cmTC_1abb2] Error 1
make[1]: Leaving directory '/home/niko/git/handle_allocator/build/CMakeFiles/CMakeTmp'
Makefile:140: recipe for target 'cmTC_1abb2/f### Other Notes/Information

<!-- Write whatever you feel is relevant -->
ast' failed
make: *** [cmTC_1abb2/fast] Error 2

Source file was:
#include <pthread.h>

static void* test_func(void* data)
{
  return data;
}

int main(void)
{
  pthread_t thread;
  pthread_create(&thread, NULL, test_func, NULL);
  pthread_detach(thread);
  pthread_cancel(thread);
  pthread_join(thread, NULL);
  pthread_atfork(NULL, NULL, NULL);
  pthread_exit(NULL);

  return 0;
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/niko/git/handle_allocator/build/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_399cb/fast && /usr/bin/make  -f CMakeFiles/cmTC_399cb.dir/build.make CMakeFiles/cmTC_399cb.dir/build
make[1]: Entering directory '/home/niko/git/handle_allocator/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_399cb.dir/CheckFunctionExists.c.o
/usr/bin/gcc-7   -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_399cb.dir/CheckFunctionExists.c.o -c /usr/share/cmake-3.19/Modules/CheckFunctionExists.c
Linking C executable cmTC_399cb
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_399cb.dir/link.txt --verbose=1
/usr/bin/gcc-7  -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic CMakeFiles/cmTC_399cb.dir/CheckFunctionExists.c.o -o cmTC_399cb  -lpthreads
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
CMakeFiles/cmTC_399cb.dir/build.make:105: recipe for target 'cmTC_399cb' failed
make[1]: *** [cmTC_399cb] Error 1
make[1]: Leaving directory '/home/niko/git/handle_allocator/build/CMakeFiles/CMakeTmp'
Makefile:140: recipe for target 'cmTC_399cb/fast' failed
make: *** [cmTC_399cb/fast] Error 2

Platform and Versions

Operating System: Ubuntu 18.04
CMake Version: 3.10.2 and 3.19.3
VSCode Version: 1.52.1
CMake Tools Extension Version: 1.5.3
Compiler/Toolchain: GCC 7.5 (C++17)

Set cmake.loggingLevel to "Debug" in your settings. Then reload the project and let it configure. Share the output. The above is only a build output, right? I see no configure step there.
Then also paste the exact command line that succeeds in the terminal, so that we compare. Hopefully we can identify some difference in arguments given to CMake, or a different folder where it's invoked...

Hi, thanks for the answer. :smile: I have cmake failing on some other similarly configured project in general so it seems there is some issues on my side. I'll check and report back.


chore(frontend): Remove Unused Svelte Files

The following files were removed because they were not used in the project anymore:
$(cat deleted_files.txt)

Testing workflow

Test failed


Zombie records?

I keep deleting these two records: https://geodev.btaa.org/?f[b1g_genre_sm][]=Geospatial+data&q=&search_field=all_fields
I first deleted them from GEOMG a few weeks ago. Then, I manually deleted them from Solr. They will initially disappear from Solr, but then show up again a few minutes later. I thought it had to do with the lag in the GEOMG update time. However, it still occurs now even though I removed them from GEOMG long ago. I have successfully deleted other items in the same way, but these two keep reanimating.

After wiping Solr, this is no longer relevant. Not neccessarily solved, but closing for now.


modeset for tab collapsing

First up: GREAT plugin... love it!!!
Second, is there a way to modeset the tabs collapsing? I use vim-workspace and would like to have my tabs span across the top as opposed to being collapsed into one square display on the left.
Is this a bug? Should the tabs span anyway? I uninstalled vim-workspace but no change, just lose tab navigation functionality.
Thanks

Ah sorry about that, he only recently changed the name.
Perfect!! Disabling the tabline in your plugin was all I needed :) thanks a bunch!!


Add Swift Package Manager support

Now that Xcode 11 supports Swift Package dependencies, it would be interesting to add the possibility to use this repository as Swift Package.

Bump! Please!

Because this library's Pod install depends on Alamofire base pod, I can't easily use Swift Package Manager for Alamofire and Pods for just this library until it is converted (since it will install another copy of Alamofire on top of the SwiftPM version).   This can be managed temporarily with a local podspec file like this, but that's really kludgy.


Error when loading content with Blob URL in player.load()

Have you read the Tutorials?
Yes
Have you read the FAQ and checked for duplicate open issues?
Yes
If the question is related to FairPlay, have you read the tutorial?

N/A
What version of Shaka Player are you using?
4.7.11
What browser and OS are you using?
OS: Windows 10 Version 22H2
Browser: Google Chrome | 122.0.6261.131 (Official Build) (64-bit)
Please ask your question
I encountered an issue while attempting to load content using a Blob URL with Shaka Player's player.load() function. When trying to load content using a Blob URL generated from a modified MPD file, I encountered an error that resulted in the player failing to load the content properly.
Code Snippet:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>Download and Play</title>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/FileSaver.js/2.0.0/FileSaver.min.js"
    integrity="sha512-csNcFYJniKjJxRWRV1R7fvnXrycHP6qDR21mgz1ZP55xY5d+aHLfo9/FcGDQLfn2IfngbAHd8LdfsagcCqgTcQ=="
    crossorigin="anonymous" referrerpolicy="no-referrer"> </script>

  <script src="https://cdnjs.cloudflare.com/ajax/libs/shaka-player/4.7.11/shaka-player.compiled.js"></script>
</head>
<body>
  <input type="text" id="urlInput" placeholder="Enter URL">
  <button id="downloadBtn">Download & Play</button>
  <video id="video" width="640" controls></video>
  <script>
    const shaka = window.shaka;
    const video = document.getElementById("video");
    const player = new shaka.Player(video);

    video.addEventListener("play", () => {
      player.configure({
        streaming: {
          rebufferingGoal: 10,
          bufferingGoal: 10
        }
      });
    });

    shaka.net.NetworkingEngine.registerScheme('blob', shaka.net.HttpFetchPlugin);

    document.getElementById("downloadBtn").addEventListener("click", function () {
      var url = document.getElementById("urlInput").value;

      // Modify the extension ID as needed
      var id = "jgefhgchodfjnjemnclijejgnlifdbbg";

      chrome.runtime.sendMessage(id, { url: url }, function (response) {
        console.log("Response from extension:", response.fileContents);
        mpd = response.fileContents.replace("input_000_dashinit.mp4", "https://github.com/azoozs/shaka-player-multi-period/raw/main/videos/input_000_dashinit.mp4 ")
                                   .replace("input_001_dashinit.mp4", "https://github.com/azoozs/shaka-player-multi-period/raw/main/videos/input_001_dashinit.mp4 ")
                                   .replace("input_002_dashinit.mp4", "https://github.com/azoozs/shaka-player-multi-period/raw/main/videos/input_002_dashinit.mp4 ");
        console.log("Modified MPD content:");
        console.log(mpd);

        var blob = new Blob([mpd], { type: "application/dash+xml" });
        var urlblob = URL.createObjectURL(blob);
        player.load(urlblob, null, 'application/dash+xml');
      });
    });
  </script>
</body>
</html>

Try entering the following URL https://raw.githubusercontent.com/azoozs/shaka-player-multi-period/main/videos/out.mpd into the input field and observe the behavior.
Error:
networking_engine.js:567 Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'timeMs')
    at networking_engine.js:567:20
    at ff (abortable_operation.js:215:19)
    at abortable_operation.js:187:47

I don't believe we should throw that exception no matter what.  That may be a bug in NetworkingEngine, or it may be an incompatibility between HttpFetchPlugin and blob URLs, which is was not explicitly designed to handle.
However, I see a separate fundamental issue with your snippet.  GitHub does not give you permission to load that mp4 content from all origins:
$ curl -sD /dev/stderr https://github.com/azoozs/shaka-player-multi-period/raw/main/videos/input_000_dashinit.mp4 > /dev/null
HTTP/2 302
server: GitHub.com
date: Fri, 29 Mar 2024 17:54:25 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
access-control-allow-origin: https://render.githubusercontent.com

The only origin other than github.com that is permitted to load that file is render.githubusercontent.com.  Your browser will not fulfill that request from any other origin due to CORS (Cross-Origin Resource Sharing) rules.  You can find more on that topic here: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

@joeyparrish
Thank you for your prompt reply. Could you please provide a comprehensive example of how to download an .mpd file from a Blob URL?
Regarding the CORS issue, I'll explore alternative hosting options and test the solution.

I was wrong about blobs.  The fetch plugin is explicitly registered for blobs already:
https://github.com/shaka-project/shaka-player/blob/12bf6428e00b93cc2cf9cefcf4db39fe9c749ee3/lib/net/http_fetch_plugin.js#L331-L334
So it should be able to handle them.  You shouldn't need to register anything at all, just load a blob URL with an explicit MIME type in player.load():
https://shaka-player-demo.appspot.com/docs/api/tutorial-blob-url.html
player.load(blobURL, startTime, 'application/dash+xml');

If that doesn't work, please file an issue using the bug template and provide an explicit example of a blob that triggers the exception.  Thanks!

Thank you, @joeyparrish, for your prompt and insightful assistance. For anyone facing the same problem in the future, here is the solution:
1- Make sure that you specify the absolute path to each .
2- Ensure that there are no small gaps between the periods (you can use my Python code to create a .mpd file without small gaps between the periods).
3- Remove this line:
shaka.net.NetworkingEngine.registerScheme('blob', shaka.net.HttpFetchPlugin);

After these steps, player.load should work properly.
Full code (you can test by entering: https://raw.githubusercontent.com/azoozs/shaka-player-multi-period/main/example_works_fine/input_001_dash.mpd):
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>Download and Play</title>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/FileSaver.js/2.0.0/FileSaver.min.js"
    integrity="sha512-csNcFYJniKjJxRWRV1R7fvnXrycHP6qDR21mgz1ZP55xY5d+aHLfo9/FcGDQLfn2IfngbAHd8LdfsagcCqgTcQ=="
    crossorigin="anonymous" referrerpolicy="no-referrer"> </script>

  <script src="https://cdnjs.cloudflare.com/ajax/libs/shaka-player/4.7.11/shaka-player.compiled.js"></script>
</head>
<body>
  <input type="text" id="urlInput" placeholder="Enter URL">
  <button id="downloadBtn">Download & Play</button>
  <video id="video" width="640" controls></video>
  <script>
    const shaka = window.shaka;
    const video = document.getElementById("video");
    const player = new shaka.Player(video);

    video.addEventListener("play", () => {
      player.configure({
        streaming: {
          rebufferingGoal: 10,
          bufferingGoal: 10
        }
      });
    });

    //shaka.net.NetworkingEngine.registerScheme('blob', shaka.net.HttpFetchPlugin);

    document.getElementById("downloadBtn").addEventListener("click", function () {
      var url = document.getElementById("urlInput").value;

      // Modify the extension ID as needed
      var id = "jgefhgchodfjnjemnclijejgnlifdbbg";

      chrome.runtime.sendMessage(id, { url: url }, function (response) {
        console.log("Response from extension:", response.fileContents);
        let mpd = response.fileContents;

        console.log("Original MPD: ", mpd);

        // Create Blob Object
        const blob = new Blob([mpd], { type: "application/dash+xml" });
        const urlblob = URL.createObjectURL(blob);
        player.load(urlblob, 0, 'application/dash+xml');
      });
    });
  </script>
</body>
</html>


Fix tag update logic for EKS-A Prowjobs

Without the dash, the entirety of the tag was being selected for regex grouping, which caused an incorrect substitution.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve


Prevent database transaction when BeforeUpdate/Delete raises Exception

Would there be a way to disable final database transaction if e.g. the BeforeUpdate/Delete raises an exception?
This would be a nice way to use the 'BeforeXXX' events as kind of a request of a user to ask for changes that need to be approved by another user.
If these Events had a 'needApprove' property, nice request/grant workflows could be created with this package. An approval could then apply the BeforeXXX event to the database.
Many thanks for your opinion and reply.

@w4mike this is an interesting use case. The problem with disabling the final DB transaction like you mentioned means it would also not store the event since triggers run within the same transaction.
However, what might be possible is overriding the Before... trigger to RETURN NULL; at the end of the trigger. You'd have to do this by calling super().get_func(model) and then replace the return statement with the proper value. This would tell Postgres to ignore that operation from happening, but it would still store the event. You'd still have to figure out how to propagate this information to a user to let them know something needs approval.
I don't think you need a needApprove property for the event either - You could just treat the event table as a list of events that need processing and then have another model that has a 1-1 reference to the event to mark it as processed. Not ideal, but it's one way of accomplishing it.
I'm planning on doing an overhaul on pghistory to make it more flexible for cases like this. Here are some things I'm planning to do that might help out your use case:

I'm planning to make the base model for events more flexible. Although you can currently make a custom event model, I don't think you'd be able to add in a custom boolean field with a default value. I'm planning to address this
I'm going to consider making the RETURN statement configurable so that you can RETURN NULL and block the operation while still writing the event.

I'm going to mark this as a question and leave it open until I have some of these features in place.


Fix SF10a driver. Add support for SF11c and rename to SF1xx

Current i2c driver for SF10a (also for all SF1XX family) is non-functional and violates device datasheet.  I've fixed it and added support for SF11C altimeter

Pretty cool, thanks!

@jgoppert I've added SENS_EN_SF1XX with following possible values:

0 - disable
1 - SF10a
2 - SF10b
3 - SF10c
4 - SF11c

Also i've fixed long standing bug with missed orb_unadvertise on driver stop which leads to broken publish after driver stop/start cycle. Also there no close calls to opened fd's.
I've found that many of the device drivers forgets to:

orb_unadvertise on stop (such driver will broken after restart).
close file descriptors.

If you have time to do a quick flight with the driver change I would appreciate it. I have a vehicle flying with an sf10a and don't want to find a glitch in flight.

I'll try to check it in hands (without props) today.
Also please do ground test too to be sure that sf10a also stream measurements (its datasheet say only few words about i2c protocol)

on the ground:
http://logs.uaventure.com/view/oWJUdudR9eQzFy7nMCjhrd
in the hands:
http://logs.uaventure.com/view/ZYSs7xfuXFxZnqvN3uAYWe

@svpcom I am curious whether the readings are correct. This is what your first log (on the ground) shows as raw readings:

http://logs.uaventure.com/view/ZWrto8kKKyszWzNswzQuyN
this is log from my today's flight

is the second plot from sf10a lidar or it is from sf11c? sf11c is a long-range and has more fluctuations in short range measures.

@ecmnet Can't you compare old and new sf1xx drivers on sf10a lidar? I'm unable to compare old because it is not work on my sf11c.

@ecmnet Also sf11c has builtin median filter (disabled by default). I've enabled it and it removed spikes. So now it report altitude with +- 1cm deviance

@svpcom I will do a comparison as soon as I finde some time. Some experiments with SF10a two years ago showed, that I2C communication (e.g. the clock speed and EMV) could influence the device showing random or repeating spikes. Could you share a latest log?

@ecmnet
http://logs.uaventure.com/view/8kBhiED5fLrqaDpoZLNRmN
sf11c with internal median filter and speed vs distance measurement mode.
max flight altitude was ~ 150m, terrain - high grass.

It seems i've found issue of spikes -- i2c address collision with rgbled driver

@svpcom What configuration and wiring is required on the sensor side to make this work? Can you share a picture? Ideally you would send a PR for PX4/Devguide.

Before this you should connect to sensor via usb (it has internal usb2serial converter),  run terminal, press 'space' and check that i2c address == 0x66. Newer sensor versions already have 0x66 preconfigured.

@LorenzMeier Is it possible to merge into upcoming 1.4.1 release?

Generally yes, but I would love to have some cross-testing from other users in this thread.

@jgoppert , @ecmnet  Could you test new SF1xx driver with SF10a lidar?

@svpcom I've planned some tests for today.

@svpcom @LorenzMeier @jgoppert Tested the new driver with SF10a and it works fine for me. I also did a flight test with V1.4.1RC1: http://logs.uaventure.com/view/w6HFPnTK8MkeQA3HJYg6J5. It would be great to see it in V1.4.1.

@LorenzMeier Changes was cross-tested by @ecmnet, documentation already merged to devguide. I think it ready for v1.4.1 merge

It's rebased on master, thanks!


PLATFORM_RPI doesn't compile on Raspberry Pi 3 / Raspberry Pi OS Bullseye

I fixed some errors, but got stuck on this:
make PLATFORM=PLATFORM_RPI
gcc -c rcore.c -Wall -D_DEFAULT_SOURCE -DPLATFORM_RPI -DGRAPHICS_API_OPENGL_ES2 -Wno-missing-braces -Werror=pointer-arith -fno-strict-aliasing  -std=c99 -fPIC -DEGL_NO_X11 -I. -Iexternal/glfw/include -Iexternal/glfw/deps/mingw -I/opt/vc/include -I/opt/vc/include/interface/vmcs_host/linux -I/opt/vc/include/interface/vcos/pthreads
rcore.c:377:9: error: unknown type name ‘EGL_DISPMANX_WINDOW_T’
  377 |         EGL_DISPMANX_WINDOW_T handle;       // Native window handle (graphic device)

The problem is the same as this: https://forums.raspberrypi.com/viewtopic.php?t=311687
The answer given there is that Bullseye no longer supports this GL stack, and that DRM should be used instead.  So if this is never going to work with any future versions, should PLATFORM_RPI be removed or marked as depreciated from Raylib?

I'm not running the experimental 64-bit Raspberry Pi OS, just the standard 32-bit, but still my usr/include/EGL/eglplatform.h doesn't include EGL_DISPMANX_WINDOW_T and /opt/vc/include/EGL/eglplatform.h doesn't exist, so it seems they have removed it from all versions (?)

PLATFORM_DRM works on the Pi 3.

PLATFORM_RPI requires Dispmanx libraries and they have been removed on RPI OS Bullseye. This is a known issue.
PLATFORM_RPI is left for RPI OS Buster users, that still use Dispmanx. Maybe it will be removed in the future.
This issue has been commented in https://github.com/raysan5/raylib/issues/2229, also mentioned in raylib Discord many times...
It probably needs to be better documented in rcore.c, in Makefile and the raylib Wiki.

Yeah I think the wiki page should make it clear that if you're on Bullseye you should use PLATFORM_DRM, regardless of what model Pi you have.

I updated wiki a little.

I further reviewed the Wiki: https://github.com/raysan5/raylib/wiki/Working-on-Raspberry-Pi
Probably PLATFORM_RPI will be removed in the future.


Read individual frames of multi-frame DICOM image

In case of multi-frame images, ImageFileReader loads the entire pixel data element into memory at once when executed. However, this may not be desired under certain circumstances, for example if the image is very large. It would thus be great if one could read only a subset of frames from a file.
Is this supported in a way that I missed or is there a workaround?
It looks like GDCM supports reading parts of the pixel data element via StreamImageReader and/or ImageRegionReader. However, this functionality doesn't seem to be exposed as part of ITK's GDCMImageIO.

Adding streaming support to SimpleITK's ImageFileRead is on the todo list. But I am not certain what the support of the GDCMImageIO is for this. I recommend asking the ITK comminity if GDCMImageIO supports this feature on the ITK forum: https://discourse.itk.org

There is an initial implementation of streamed reading for SimpleITK here #468. Additionally the ability to specify the which ImageIO to use for reading is added here, and that has been merged into the next branch for testing.
With these features implemented, if ITK supports this format than SimpleITK will. Additionally, if you compile SimpleITK with DCMTK over GDCM you can select that with the set ImageIO feature.

With the patch #468 and the updated GDCM in the ITK release this feature should not be reasonably implemented.


Fix default histogram max.

Revert histogram max to 5M (was typo'd to 500k in commit a6b679d5.

+1 merge - resolves failing eunit test for me.


Fix symlink to IDEA installation directory

This closes issue #12 upstream.

Hey @l0b0,
thanks for your PR. Would you be willing to fix the errors that cause Travis to fail?
Cheers!
Daniel

Do you have a link to the issue? All I could see were puppet-lint issues
not related to my change.
Cheers
Victor
On Mon, Jul 6, 2015 at 10:40 AM, Daniel Kerwin<EMAIL_ADDRESS>wrote:

Hey @l0b0 https://github.com/l0b0,
thanks for your PR. Would you be willing to fix the errors that cause
Travis to fail?
Cheers!
Daniel
—
Reply to this email directly or view it on GitHub
https://github.com/gini/puppet-idea/pull/14#issuecomment-118787612.

Do you have any update on this?

Ping?

I'm no longer using puppet-idea (or Puppet for that sake, Nix FTW :wink:), and this looks like it's not going to get merged, so I'll just close it for neatness. Feel free to reopen if you want to merge it.


Initial pass at functionality

not sure if you meant to request review yet but looks good so far 👍


Add for waterdogPE

Description

Add luckperms for waterdogPE proxy.
Proposed behaviour

Nothing.

https://github.com/lucko/LuckPerms/issues/2876


Changed way to include PVRTexTool library.

It didn't work in my system the way it was right now. Also, this is the way Qt docs states.

fixes #70

Thanks!


Add goal inference and time series projects

Motivation
Since there are fewer BM maintainers within Meta, I wanted to open source intern infra projects (by James and Zhen) built on BM for external research and future use. These are all implementations of publicly available papers and contain no internal data.  I would like to get this in if tests pass and defer design and functionality questions to a future PR since these things are a bit time sensitive.
Test Plan
I've run this internally, but I'm having issues installing BM on my laptop now (I'll open a separate issue about that) so I'm going to defer to CI's testing.  Note that the individual tests for the two experimental projects are not being run as part of our CI suite but since they are experimental, that is not a blocker atm.

Sweet! This looks really cool

Thanks for the review @ndmlny-qs I'll fix them this week. Good catch on the imports, I now notice pyre is not being run as part of my old linter since I switched devservers.

I guess this is really a question around legal; since BM is strictly OSS and these projects were previously in BM but not pushed to github, it is more straightforward to make them a part of the project (since BM is already approved for that), and then spun off to their own libraries rather than going through the internal process to oss a new project under Meta's organization which IIRC took months with coordination across multiple pocs. WDYT?


Potential security issues with get_ip() changes in v4

I'm a little concerned by the change to use ipware.ip.get_ip() in v4.

There is no documentation to suggest that django-ipware may need to be configured.
There is now no documentation on the potential attack vector if the proxy doesn't set X-Forwarded-For (or whatever the first header used in IPWARE_META_PRECEDENCE_ORDER is) explicitly, c.f. USE_X_FORWARDED_HOST
django-ipware says to use get_ip() if the server is not accessible on the Internet, see here. (Although looking at the code the distinction between get_ip() and get_real_ip(), the latter doesn't count private IP addresses as "real". What they really mean here is use get_ip() if there are also clients that may connect from an internal private network otherwise use get_real_ip().)
Previously django-axes allowed configuring the maximum number of proxies in case of spoofed entries. This is no longer possible. django-ipware provides get_trusted_ip() but it isn't particularly useful in it's current form.

It would probably be wise to recommend setting IPWARE_META_PRECEDENCE_ORDER to ['HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR'] or at least indicate that this was the original behaviour. Checking all of these headers by default feels risky.
Forgive me, but this feels rushed and ill thought through. It was working fine for me and issue #275 and pull request #280 fail to even give any evidence of a problem. Every comment in #275 is regarding AttributeError when looking up META on NoneType - django-ipware is still using request.META. This feels like a push to remove code in favour of a dependency for minor gain.

Current state of django-axes is that it is seriously broken and an adversary can completely bypass its IP limits.
django-axes uses the deprecated API from django-ipware.
If the Django application is deployed without a reverse proxy server, django-axes via ipware blindly trusts and uses the incoming headers. As a remedy, if IPWARE_META_PRECEDENCE_ORDER is set to an empty list, then get_ip() always returns None. This, at best, has the consequence of grouping all attempts into the same bucket.
If the Django application is deployed behind a reverse proxy server, then if the client sends a header (say an adversary intentionally does this) and proxy only appends to it (behavior for many proxy servers including Apache's ProxyPass, haproxy's forwardfor and nginx's proxy_add_x_forwarded_for), then the IP address from the client sent header is used. For example:
Django project is configured as:
IPWARE_META_PRECEDENCE_ORDER=('HTTP_X_FORWARDED_FOR',),

Running under a WSGI server reverse proxied using Apache.
curl -H 'X-FORWARDED-FOR: <IP_ADDRESS>' https://<server>/

In this case, ipware.ip.get_ip() returns <IP_ADDRESS> and not the IP address of the client. Hence, by continuously changing the header one can attempt any number of logins and bypass django-axes.
I believe the right course of action is to announce to the users that current version is insecure and quickly make a release with fixes.

I think there are several aspects of ipware.ip2 that could have been done better:

Handling cases where Django project is not deployed behind a reverse proxy seems convoluted and undocumented. Caller has to set IPWARE_META_PRECEDENCE_ORDER = ('REMOTE_ADDR',) and set proxy_count = None.
proxy_count = None is different from proxy_count = 0. This only causes confusion unless explained better.  proxy_count = 0 seems to mean that there is one trusted server before the Django project.
Why have some settings such as IPWARE_META_PRECEDENCE_ORDER be global settings and others such as proxy_order, proxy_count and proxy_trusted_ips be arguments when all of them are tightly related to how reverse proxies are configured? It is inconvenient to have apps send all the method parameters when they are likely to be same throughout the app.
When there are client side proxies involved in the chain, ipware.ip2 seems to want to return None due the following code:

            if proxy_count > 0 and proxy_count != ip_count - 1:
                # we are expecting requests via `proxy_count` number of proxies
                continue

When client side proxies such as squid add 'X-Forwarded-For' and server side proxies append to it, then there is no way to properly configure the a Django project using ipware. The expected proxy_count will be different from ip_count - 1.

If the Django application is deployed without a reverse proxy server, django-axes via ipware blindly trusts and uses the incoming headers. As a remedy, if IPWARE_META_PRECEDENCE_ORDER is set to an empty list, then get_ip() always returns None. This, at best, has the consequence of grouping all attempts into the same bucket.

Looks like IPWARE_META_PRECEDENCE_ORDER = ('REMOTE_ADDR') is solution to this.

@SunilMohanAdapa please make a pull request. Thanks.

I have worked around the problem in my project by configuring the proxy to override the header sent from client.

@camilonova - please re-open this issue as the concerns raised have still not been fully addressed.

The default configuration of django-ipware is unsafe due to the full range of headers specified in IPWARE_META_PRECEDENCE_ORDER.
The lack of documentation in django-axes to warn that django-ipware and any proxy in front of django-axes need to be configured correctly.

What @SunilMohanAdapa has called a "workaround" is in fact what must be done to correctly and safely configure the proxy headers.
Putting this plainly, using a real example, IPWARE_META_PRECEDENCE_ORDER must not contain HTTP_X_FORWARDED_FOR if nginx does not do the following:
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Also note that this must be proxy_set_header and not proxy_pass_header. Without correct configuration, any client can set anything in the header bypassing any of the protection that django-axes claims to provide.
Obviously, django-ipware ought to set a safe default for IPWARE_META_PRECEDENCE_ORDER, using values such as REMOTE_ADDR which are usually automatically explicitly set by the server, but that ship has sailed. Ideally it should force you to specify the one header that you expect, and have none by default raising ImproperlyConfigured. (Ping @un33k)
As mentioned when I opened this issue, please compare to USE_X_FORWARDED_HOST in Django.
This change to use django-ipware was made in 9ecbf4cf6 by you @camilonova and, as mentioned in the last paragraph when I opened this issue, didn't actually address the issues referenced. It also removed any references to the security-related concerns in the documentation, vague as they were, when then the documentation for all of the options was ripped out.

@pope1ni Nick thank you for explaining. I understand what you mean, what do you propose to fix it?
I did change the library to use django-ipware because I think is better to use them than to code and maintain the logic in this library, which has a different purpose.
But I get your point, if you think there is a better (safer) alternative please go ahead and improve it.
Maybe this security issue should be handled by django-ipware and not by django-axes. I do support what is best for the library, even if is replacing django-ipware.

I think the correct fix is to improve documentation to support the configuration of django-ipware so that it is compatible the idea of how django-axes should function.
@pope1ni would you like to contribute the necessary documentation?

I propose the fix as follows:

ipware to introduce ipware.ip3.get_client_ip() to fix current issues.
ipware to not take any parameters other than request. The rest should be configuration. That will make the library more useful. django-axes and other users need not introduce their own configuration parameters just to pass to ipware.
ipware to default to assuming that the service is not running behind a proxy server as @pope1ni suggested. Assuming direct service by default is secure in all cases. Assuming behind proxy by default in insecure in some cases. This is similar to Django itself.
ipware to not have precedence order at all. Instead it should only allow specifying a single header to extract client address from. Allowing multiple headers opens users to security problems. It is hard to imagine a use case for accepting two headers securely. Have two proxies that send two different headers? You need to configure each one of them to reset the other header anyway, so why not configure them to send the same header? Falling back to REMOTE_ADDR when configured header value is unexpected may be okay.
ipware to not assume that ip address count in the value of header will be same as number of proxies. This assumption easily fails when clients send their own header and proxies append to the headers (default for most proxies).
ipware to thoroughly document all the settings and their security implications.
(optionally) ipware to implement more thorough handling of trusted proxies. This include consideration for checking REMOTE_ADDR itself in proxy list and performing normalized ipv6 address matching.
(optionally) ipware to parse the 'Forward' header differently according to RFC 7239.
django-axes to make no changes expect for asking users to look at ipware documentation for configuration in case the project is deployed before proxy servers. Default will be to assume that service is not running behind proxy.

That all sounds good and well but you will have to open the issues and pull requests into the django-ipware project. I think that it is unreasonable to ask for the author to implement changes to only benefit django-axes, so contributing that new django-ipware utility would be the prompt thing to do, if you want it.
django-axes and django-ipware do not share the developers or code base, and django-ipware seems pretty well designed to me, in all honesty. Parsing end user IPs is a best effort task and can not be done with 100% reliability, and hence the end user IPs shouldn't be relied on too heavily in terms of authentication. They can be used to augment the authentication flow to distinguish forged authentication requests coming from a single computer and ban that IP, but they have little use outside of that.
The solution in django-axes at this time is to improve the documentation to include instructions on configuring django-ipware in a sane fashion. This can, in my opinion, be as simple as directing the user refer to the django-ipware in the configuration documentation.

On Thursday 22 March 2018 11:47 PM, Aleksi Häkli wrote:

That all sounds good and well but you will have to open the issues and
pull requests into the django-ipware
https://github.com/un33k/django-ipware project. I think that it is
unreasonable to ask for the author to implement changes to only benefit
|django-axes|, so contributing that new |django-ipware| utility would be
the prompt thing to do, if you want it.
|django-axes| and |django-ipware| do not share the developers or code
base,

I understand. My special thanks to @un33k for proactively looking at
this issue and coming up with fixes in django-ipware.

and |django-ipware| seems pretty well designed to me, in all
honesty.

I hope the large number of issues @pope1ni and I pointed out here
convince you otherwise.

Parsing end user IPs is a best effort task and can not be done
with 100% reliability,

Determining remote address in a IP connection need not be guesswork.
When trusted proxies are properly configured, it should be possible to
accurately determine the remote address.
The current implementation (django-axes + django-ipware) makes this
guesswork by making several false assumptions. It makes properly
configuring the network harder.

and hence the end user IPs shouldn't be relied on
too heavily in terms of authentication. They can be used to augment the
authentication flow to distinguish forged authentication requests coming
from a single computer and ban that IP, but they have little use outside
of that.

Note that if None (or wrong IP) is returned for get_client_ip() due to
guesswork, it becomes possible to do denial-of-service attacks on
resources protected by django-axes. So, please don't take the issue lightly.

Please refer to the example answer @un33k has made above. There are a
number of ways to circumvent proxy count checking if any aspect of
network security has been overlooked, which is the case in many
projects.

This only happens because ipware makes the fatal mistake of not checking
REMOTE_ADDR in list of trusted proxies. REMOTE_ADDR is accurate and can
be trusted unlike HTTP headers. This is the starting point of the trust
chain. My proposed fix addresses the problems.
Also, the assumption here is that an adversary is able to make requests
to the service from within a private network after breaching network.
This is a far fetched case considering how we have problems with some
basic things.
[...]
I will try to make patch for both projects if I get some time to spare.
But looking at my backlog, I encourage others to do that.
Changes in axes or ipware, from me or axes or ipware devs, I urge you
all fix all the problems fundamentally and not close this issue off. It
would be a disservice to the users of both projects.
Thank you,
--
Sunil

Thanks a lot to @pope1ni and @SunilMohanAdapa for explaining the problem clearly. I reached the point in my roadmap where I needed to look into upgrading django-axes for the project I maintain and I had to read the code to figure out how to properly upgrade from v2 and figured out all configuration wrt proxies was actually removed making django-axes a useless brick in my company's codebase. I will look into getting some time to fix django-ipware but it is more likely that we will have to stick with the older release.

@EvaSDK and all other conversation participants, the issue you are discussing is related to your proxy configuration and django-ipware configuration, not axes being broken.
You need to, obviously, configure your reverse proxy to correctly pass / add / set headers and configure django-ipware to check those headers strictly by setting the IPWARE_META_PRECEDENCE_ORDER key if you wish to implement functional IP checking for your project. There are configuration options for that.
Any real project setup will require you to configure the reverse proxies you are using. I hope you are not blindly running your Django deployment without a correctly and thoughtfully configured reverse, at least.
The main thing to do regarding axes is to improve the documentation to explain these things to the end user, but please notice that security does not come to your project by installing an arbitrary package from the Internet, even if that package claims to check this or that stuff and make your project more secure.
You need to actually understand what the packages you use do, why they do what they do, and whether or not you should use them.
You also need to understand that discussing django-ipware package issues in django-axes issue tracker is not productive; if you wish to change how django-ipware works open an issue or PR there, or if you wish to change the axes functionality, open a PR here.

Don't worry, I think we all understand this. I guess the problem is that it is a regression from django-axes from our point of view.
For example, no amount of configuration from django-axes or django-ipware can get ip2.get_client_ip as used in https://github.com/jazzband/django-axes/blob/3a2fb808840666a59ad863325249aae4e74b9aea/axes/attempts.py#L18 to be trusted.
What's suggested by @SunilMohanAdapa  in https://github.com/jazzband/django-axes/issues/286#issuecomment-375334632 needs to be ported to django-ipware bug tracker and fixed.

How is it not secure or trusted? If you configure the following:
IPWARE_META_PRECEDENCE_ORDER = ('REMOTE_ADDR', )

You are good to go if you are not running behind a reverse proxy. ipware should not check for additional headers, and you will get the value for REMOTE_ADDR as your end user IP.
If you do have a reverse proxy, for example nginx, that sets the client header instead of appending to a header value to a chain of headers provided by the client making the request, you are also good to go:
# nginx configuration for location with http_realip_module enabled
proxy_set_header X-Real-IP $remote_addr;

# Django project configuration for ipware
IPWARE_META_PRECEDENCE_ORDER = ('HTTP_X_REAL_IP', 'REMOTE_ADDR', )

The cases where django-ipware becomes unsecure is

when you are using the default ipware IPWARE_META_PRECEDENCE_ORDER headers (or request.META values to be exact) to check the client IP and you allow a client to set and your proxy to forward those headers arbitrarily (which is how many proxy servers work out-of-the-box), or
you check against an appended header value such as X-Forwarded-For or request.META['HTTP_X_FORWARDED_FOR'] in Django terms, for which we can not currently configure the proxy_count as used by ipware at this moment. We can add a configuration flag for this to allow for accounting of proxy count as well.

https://github.com/un33k/django-ipware/blob/a71d956c1c3c21c80e742823687f39e970e1147c/ipware/ip2.py#L5
The things that can happen to your project because of this are

someone is able to make multiple authentication requests from a single IP by spoofing headers, bypassing axes lock out mechanisms, which any serious attacker apart from a script kiddy with a single computer will indeed do if they are really interested in breaching your site by e.g. using a fleet of low cost spot pricing cloud servers that have changing IPs, or
you lock out users by accident because you use e.g. the REMOTE_ADDR attribute behind a proxy which gets set to <IP_ADDRESS> and you end up seeing multiple authentication requests coming from the same address, locking that address, and all your users that seem to use the same address, out.

The case that is really potentially insecure here is case 1, where you allow your clients to effectively inject semantical headers that you use for secure checking into your service, and do not check their semantics or counts. Even after bypassing the lock out mechanisms your attacker still has to brute-force their way in, usually requiring and unrealistic amount of attempts over a slow HTTP connection over TCP/IP socket as an attack vector and a brute-force attack against (most probably) PBKDF2 hashing, which is rather slow as default. Then your attacker has to actually gain access to a meaningful account to exploit your service, meaning that you have to have a service that requires heightened security running with non-MFA staff users. After those steps, yes, an attacker can gain access to your accounts, and if you use single-factor authentication as in just a password to actually log in as a staff user in a heightened security online service, I'd say you are actually fucked by design.
Or is there something I'm missing here? Is there an easier and simpler attack vector that is open, if you have configured your proxy, ipware, and axes correctly?

I'm all for making this the default:
IPWARE_META_PRECEDENCE_ORDER = ('REMOTE_ADDR', )

that would at least provide secure defaults. (I've suggested this back in 2015: https://github.com/un33k/django-ipware/issues/8)
A good alternative solution is https://pypi.python.org/pypi/WsgiUnproxy which acts as wsgi middleware. It corrects request.META['REMOTE_ADDR'] to point to the proper address only when the request came from a trusted list of IP addresses. This not only fixes django-axes, but also all other third party package like contact forms, rate limiting and raven/sentry to read the same IP address. It makes django-ipware pretty much obsolete imho.
For AWS, there is https://github.com/LabD/wsgi-aws-unproxy
Example:
from django.core.wsgi import get_wsgi_application
from netaddr import IPNetwork
from wsgiunproxy import unproxy

application = get_wsgi_application()
application = unproxy(trusted_proxies=IPNetwork('<IP_ADDRESS>/8'))(application)

FYI,
ipware 2.0.2 allows for this ...
get_client_ip(request, request_header_order=['X_FORWARDED_FOR', 'HTTP_X_FORWARDED_FOR'])

@un33k thanks! We'll add an axes configuration with strict defaults for sending the proxy count and request header precendence order to ipware. That should solve this problem :+1:

@aleksihakli, after considering your comment (https://github.com/jazzband/django-axes/issues/286#issuecomment-380370147) and checking our stack, I think I now agree with you than it is sensible to only consider {{REMOTE_ADDR}} as the source of information for client's IP address, be it behind proxies or not, and no changes to django-axes are required.
Using {{REMOTE_ADDR}} as the single source of truth makes everyone's code easier and hopefully avoids fallbacks to broken headers by default.
For what concerns the project I work on, raven and django-axes now behave properly and I just need to patch django-analytical to use django-ipware as well or at least unset X-Forwarded-For so it does not report incorrect data.


Get percentage of the whole scratch area

Hi and thank you for this library, it's very helpful!
But I have a question, can we get the percentage or check if the whole scratch area is revealed? as for now, we only can detect the percentage of the text/image area inside the scratch view. Any help would be appreciated! Thank you very much

Dear Liauli,
I think you can calculate the same yourself. e.g

@Override
    public void onRevealPercentChangedListener(ScratchImageView siv, float percent) {
        // on image percent reveal
float scratchedOutPercentage = 100-percent;
    }

Hope it helps you.

Hello gauravarora90, I tried the code you posted, but it is not showing the percentage on the textview.
I am trying to make a game where the player has to scratch to reveal a picture, and I want to allow him to only scratch a certain percentage of the picture.
Here is my code:
private float scratchedOutPercentage;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    ScratchImageView scratchImageView = new ScratchImageView(this);

    final TextView percentageTV = (TextView) findViewById(R.id.scratchPercentageTV);

    scratchImageView.setRevealListener(new ScratchImageView.IRevealListener() {
        @Override
        public void onRevealed(ScratchImageView tv) {
            // on reveal
        }

        @Override
        public void onRevealPercentChangedListener(ScratchImageView siv, float percent) {
            // on image percent reveal
            scratchedOutPercentage = 100-percent;

        }
    });

    percentageTV.setText(String.valueOf(scratchedOutPercentage));
}`

Hi @subzeroz ,
Change your code to put setText inside reveal listener callback.
Something equivalent to below.
private float scratchedOutPercentage;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    ScratchImageView scratchImageView = new ScratchImageView(this);

    final TextView percentageTV = (TextView) findViewById(R.id.scratchPercentageTV);

    scratchImageView.setRevealListener(new ScratchImageView.IRevealListener() {
        @Override
        public void onRevealed(ScratchImageView tv) {
            // on reveal
        }

        @Override
        public void onRevealPercentChangedListener(ScratchImageView siv, float percent) {
            // on image percent reveal
              scratchedOutPercentage = 1-percent; // Note : percent value will be between 0.0 to 1.0

            percentageTV.setText(String.valueOf(scratchedOutPercentage));

        }
    });

}

Thanks,
Harish

Hi @sharish ,  @subzeroz
I tried your code but didn't work. I had to change
ScratchImageView scratchImageView = new ScratchImageView(this);
with
ScratchImageView scratchImageView = findViewById(R.id.ScratchImageView);


Add potential dupe otk upload test

main app: receive to-device event from a new user which will cause a new OTK to be uploaded
we intercept that OTK upload request and delay it
we then open the nse process who will also see the same to-device event because the to-device since token was not incremented yet (which is correct)
if there's a bug, the nse process will upload a different OTK which we will not block.
then we unblock the main app OTK upload request.
=> dupe otk upload.

Locking seems to work correctly here, I can't break it. Need to clean up the test before PRing though.

This got added and it turns out it is helping to find the x-process lock issue.


Update document requires atomic operators

i run cluster example with MongoDB, broker can handle publish with NOT retain flag but when i set Retain flag true, it rises this error;
/home/ioboard/mqttbroker/examples/clusters/node_modules/mongodb/lib/bulk/common.js:647
throw new TypeError('Update document requires atomic operators');
^
TypeError: Update document requires atomic operators
at FindOperators.updateOne (/home/ioboard/mqttbroker/examples/clusters/node_modules/mongodb/lib/bulk/common.js:647:13)
at executeBulk (/home/ioboard/mqttbroker/examples/clusters/node_modules/aedes-persistence-mongodb/persistence.js:200:38)
at MongoPersistence.storeRetained (/home/ioboard/mqttbroker/examples/clusters/node_modules/aedes-persistence-mongodb/persistence.js:185:3)
at PublishState.storeRetained (/home/ioboard/mqttbroker/examples/clusters/node_modules/aedes/aedes.js:166:29)
at makeCallTwo (/home/ioboard/mqttbroker/examples/clusters/node_modules/fastseries/series.js:150:3)
at release (/home/ioboard/mqttbroker/examples/clusters/node_modules/fastseries/series.js:138:7)
at resultList (/home/ioboard/mqttbroker/examples/clusters/node_modules/fastseries/series.js:125:3)
at Aedes.series [as _series] (/home/ioboard/mqttbroker/examples/clusters/node_modules/fastseries/series.js:45:7)
at Aedes.publish (/home/ioboard/mqttbroker/examples/clusters/node_modules/aedes/aedes.js:257:8)
at Client.enqueuePublish (/home/ioboard/mqttbroker/examples/clusters/node_modules/aedes/lib/handlers/publish.js:60:21)
Worker 204140 died with code: 1, and signal: null
Starting a new worker
Worker 206756 is online
what's wrong here?
System Information

Aedes: 0.44
NodeJS: v10.19.0
OS: Ubuntu 20
Arch: x86_64

@KevinVitale IMO the problem is that upsert is not atomic and could create errors in cluster enviroments here:
https://github.com/moscajs/aedes-persistence-mongodb/blob/master/persistence.js#L269
I think you could give a try by using http://mongodb.github.io/node-mongodb-native/4.0/classes/bulkoperationbase.html#handlewriteerror


specifiying dependencies on node/npm

node: >=5.7.0,
npm: >=3.0.0

https://github.com/Microsoft/pxt-microbit requirements need an update (README.md)

I propose README.md on pxt-microbit refers to requirements of pxt which should state node/npm etc in README.md.
I have referred to requirements of pxt-mixcrobit from my version of pxt-microbit-electron README.md

Pull request for pxt-microbit created
https://github.com/Microsoft/pxt-microbit/pull/304

And for pxt
https://github.com/Microsoft/pxt/pull/716


public API: Remove needless const from scalar types

Some function declarations in public headers needlessly use  const with scalar types.
This PR cleans them up, updating also their documentation (as far as it exists) and their decls in implementation files.
Targeted for 4.0 because syntactically (though not semantically) this is an API break.
Checklist

[x] documentation is added or updated

The ones in the headers are fine.  Do we want to remove the ones in the C functions?  They are doing something, albeit minor.

Do we want to remove the ones in the C functions? They are doing something, albeit minor.

Good point.
I'm open to restore them in the C implementations, though practically this would make no difference for the given functions.
What do others think?


[AutoRelease] t2-reservations-2022-09-15-66688(Do not merge)

https://github.com/Azure/sdk-release-request/issues/3175
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=1024

issue link:https://github.com/Azure/sdk-release-request/issues/3175

PR looks good, please proceed


bugfix

Is it a bug ?

This is what lint says in the travis build:
httpcache/httpcache.go:115: suspect or: req.Method != "GET" || req.Method != "HEAD"

Still this part of the library probably needs to be nuked because we don't use it and there are better libraries that do this.


Use 📒 for documentation

✏ Proposed change
Use 📒 for documentation, instead of 🌮.
🤔 Rationale
Paperback documentation often comes in a binder, it seems natural to me.
Pros: Makes more sense than a taco.
Cons: With the other syntax proposals, all the books might become confusing. The emoji apparently doesn't always look like a binder.
Alternatively we can just assume that multiline comments placed before a method, class, etc are documentation.
🕺Example
📒 This class describes every property of a banana. 📒
🐇🍌🍇
  👴 ...
🍉

I’d vote for 📗 or 📘. 📒 is described as a "ledger" whereas 📗 or 📘 are textbooks. Textbooks make more sense for documentation, as documentation should be designed to learn something.


[BugFix] push worker thread can not pop NORMAL Priority task (#7628)

In the current implementation, the push worker thread can only pop the HIGH Priority tasks, it can not pop the NORMAL priority tasks.
I fix this bug in this pr, also I split the delete_workers from the _push_worker_thread_callback, I add a stand-alone function _delete_worker_thread_callback for delete_workers

[FE PR Coverage Check]
:heart_eyes: pass : 0 / 0 (0%)

[FE PR Coverage Check]
:heart_eyes: pass : 0 / 0 (0%)