[FEATURE] Migration from Pages Router to App Router in Next.js

Detailed description
I would like to propose a migration from the existing Pages Router to the new App Router in Next.js. This transition would bring numerous benefits and enhance the overall development experience.
Benefits
Allow me to outline the advantages of adopting the App Router over the Pages Router:

Improved Performance: The App Router introduces optimized client-side routing, resulting in faster navigation between pages. By utilizing dynamic imports, it enables code splitting, leading to smaller initial page loads and improved performance for users.

Enhanced Developer Experience: The App Router simplifies the management of complex routing scenarios. It offers a more intuitive and declarative API, allowing developers to define their routes in a centralized configuration file. This approach improves code maintainability and reduces duplication, leading to a cleaner and more scalable codebase.

Enhanced Code Splitting: The App Router allows for granular control over code splitting, enabling us to dynamically load only the necessary components for each page. As a result, the initial page load is significantly faster, and subsequent navigations are more efficient.

Advanced Route Handling: The App Router introduces additional features such as nested routes, parameter-based routing, and dynamic route segments. These capabilities enable more flexible and dynamic page structures, making it easier to handle various URL patterns and create dynamic routes that adapt to different contexts.

SEO-Friendly: The App Router supports server-side rendering (SSR) and static site generation (SSG), making it SEO-friendly out of the box. It ensures that search engines can crawl and index your application's pages effectively, improving discoverability and organic traffic.

Layout Flexibility: Unlike the Pages Router, which enforces a strict file-based layout structure, the App Router allows us to have more flexibility in designing layouts. We can easily create shared layouts that wrap multiple pages without duplicating code, promoting better code organization.

Possible implementation
Migration Plan
To ensure a smooth transition from the Pages Router to the App Router, I suggest the following migration plan:

Conduct a thorough code analysis and identify all the relevant routes currently implemented with the Pages Router.

Create a test branch to implement the App Router in the existing codebase, while keeping the Pages Router intact to avoid disrupting the application's functionality during the migration process.

Migrate one route at a time, testing each page thoroughly to ensure functionality remains consistent.

Update any custom code associated with navigation, such as Link components, to work with the App Router.

Conduct rigorous testing, including performance tests, to benchmark the improvements achieved through the migration.

Ensure the codebase adheres to the best practices and conventions recommended by Next.js for App Router implementation.

Update the documentation and provide clear instructions for developers on how to work with the new App Router.

Conclusion
I believe that migrating to the App Router is a valuable step for our project, bringing numerous benefits to both developers and end-users. The improved performance, enhanced developer experience, advanced route handling, SEO-friendliness, and layout flexibility make it a compelling choice. With the suggested migration plan, we can ensure a smooth and successful transition. I would be more than happy to assist with the migration process and provide any necessary support.
Thank you for considering this proposal. I look forward to discussing and implementing this change.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hey @Deepu178
I would like to work on this under GSSoC.

Thanks @s2sharpit for creating the issue and I think is issue that need thorough understanding of it. Would like to join a quick meeting for the things you want to do?

Thank you, @Deepu178, for your response and willingness to join a meeting. I appreciate your interest in discussing the issue further. I believe a meeting would be beneficial to delve into the details and gain a better understanding of the problem.
I'm available to schedule a quick meeting to discuss the things I want to do. Please let me know your preferred time and method of communication for the meeting. I look forward to our discussion and working together to address this issue. Thank you again for your support.

https://discord.gg/fCaC8BYx?event=1125877754731180042
@s2sharpit  join the discord and we can discuss on 8:30 PM tomorrow if  you are available

https://discord.gg/fCaC8BYx?event=1125877754731180042 @s2sharpit join the discord and we can discuss on 8:30 PM tomorrow if you are available

Hey @Deepu178
I am travelling today, so can we have a meeting tomorrow whenever you are available.

sure @s2sharpit

Hey @Deepu178
Can we have meeting today, What do you think?

@s2sharpit I'm available today so we can do meeting today.

@s2sharpit I'm available today so we can do meeting today.

Can we have meeting at 8:00PM ?

Are you on discord server of CollabLearn?

Are you on discord server of CollabLearn?

Yes

Hello @Deepu178, I hope you're doing well. I wanted to follow up on our previous discussion about migrating the website from the page router to the app router. Have you had the opportunity to thoroughly evaluate the benefits of the app router in comparison to the page router? I'm eager to know if you have reached a decision regarding the migration. Thank you for your time and consideration!"


Production redirect_uri is always localhost - [next-auth][warn][NEXTAUTH_URL]

Environment

Operating System: Linux
Node Version:     v16.15.1
Nuxt Version:     3.4.2
Nitro Version:    2.3.3
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      ssr, app, plugins, runtimeConfig, build, modules, auth, sitemap, css, googleFonts, elementPlus, vite, routeRules
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
https://discord.com/oauth2/authorize?client_id=981994406955581492&scope=identify email&response_type=code&redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fapi%2Fauth%2Fcallback%2Fdiscord&state=bWw3hxRDJHeVVrgTkXh8hRZXI3U1KygkfNT6M1OgaN8
Describe the bug
PM2 configuration
module.exports = {     apps: [       {         name: 'Website',         exec_mode: 'cluster',         instances: 'max',         port: '8800',         script: './.output/server/index.mjs',         log_date_format: "YYYY-MM-DD HH:mm Z",         combine_logs: true,         log_file: '/home/ubuntu/xxx/_LOGS/WEBSITE/logs.log'       }     ]   }
Nuxt config auth conf:
auth: {         computed: {             origin:'https://xxxxx.tv'         },         provider: {             type: 'authjs',         }     },
Additional context
Im using ^0.6.0-beta.4
Before open this issue I read everything I found on every git/stackoverflow ect I tried everything but nothing.
The only way I found to not receive the error "AUTH_NO_ORIGIN" has been place the "origin" under the "computed" section, but still the redirect_uri is on localhost.
I tried to set the env variables like docs, then I tried with the "NUXT_" prefix, then I tried to inject by pm2, then I tried to add the env variable to the runtimeConfig but nothing... everything I do the redirect_uri is on localhost and I receive always the warn:
[next-auth][warn][NEXTAUTH_URL]
I spent more then 5 hours on this error, someone can help me please?
Logs
No response

Try to use env conf in your pm2 config
   {       name: 'example',       port: '3001',       exec_mode: 'cluster',       instances: 'max',       script: './lk/server/index.mjs',       env: {         "NEXTAUTH_URL": "https://example.com/api/auth",         "AUTH_ORIGIN": "https://example.com/"       }     }

I've come across a similar issue that stemmed from not configuring the NEXTAUTH_URL. While going through the documentation (link), it wasn't immediately apparent that setting this variable was necessary. I had assumed that defining the origin and the default basePath ('/api/auth') would suffice.
It's worth noting that this behavior might be disrupted due to the error detailed in this GitHub issue: sidebase/nuxt-auth#515.
After some troubleshooting, I found that the only reliable setup for now is to establish both the NEXTAUTH_URL and AUTH_ORIGIN environment variables in a production environment.

I've come across a similar issue that stemmed from not configuring the NEXTAUTH_URL. While going through the documentation (link), it wasn't immediately apparent that setting this variable was necessary. I had assumed that defining the origin and the default basePath ('/api/auth') would suffice.
It's worth noting that this behavior might be disrupted due to the error detailed in this GitHub issue: sidebase/nuxt-auth#515.
After some troubleshooting, I found that the only reliable setup for now is to establish both the NEXTAUTH_URL and AUTH_ORIGIN environment variables in a production environment.

I've faced the same issue and setting NEXTAUTH_URL alongside AUTH_ORIGIN worked. Thanks @dhondtlaurens!

Yes, setting both is currently only the way to properly make this work. Does someone want to update the docs to make this more clear, otherwise ill put it on my TODO!


node: No such file or directory?

On Ubuntu 14.04:
$ speed-test
/usr/bin/env: node: No such file or directory
$

Did you install it with the global flag? npm install -g speed-test

No, but I just reran with the global flag. No change.
On Thu, Aug 20, 2015 at 4:48 PM, Frederik Krautwald <
<EMAIL_ADDRESS>wrote:

Did you install it with the global flag? npm install -g speed-test
—
Reply to this email directly or view it on GitHub
https://github.com/sindresorhus/speed-test/issues/10#issuecomment-133166425
.

http://bfy.tw/1PGx

Fixed it.
$ sudo ln -s "$(which nodejs)" /usr/bin/node
$


Move callback creation into plans

In some cases, a plan may need to do something to get information before callbacks are instantiated. Also, callbacks can be specific to the plan. therefore, they should be managed by the plan, probably in get_plan

Duplicate of https://github.com/DiamondLightSource/python-artemis/issues/656


Cannot replace root elements in html-fragment

Paragraph 1

Paragraph 2

Paragraph 3

As there is no root, how can I search the p and replace it?
I tired to use filter and replace, but it will remove all my matched elements.
$dom = FluentDOM::QueryCss('
Paragraph 1

Paragraph 2

Paragraph 3
', 'html-fragment');
$dom->filter('p')->first()->each(function($element) {
	$element->replace('hi');
});
echo $dom;

Expected:
hi
Paragraph 2

Paragraph 3

Actual:

Paragraph 2

Paragraph 3

If you're loading an HTML fragment the top level elements should be automatically selected. Here is a problem with replace() however.
At current implementation required the node to have a parent that is an element node. But in this case it is the document node. I added a test and a fix.

I have tried the new commit. filter seems can search deeply.
Code:
$dom = FluentDOM::QueryCss('
5
4

', 'html-fragment');
$dom->filter('p')->each(function($element) {
	$element->replace('hi');
});
echo $dom;

Expected:

5
4

Actual:
hi

In jquery:
var test = $("
5
4

");
test.filter("p").replaceWith('hi');
console.log(test.outerHTML());

Result:

5
4

And with the following Code, the result is also wrong:
$dom = FluentDOM::QueryCss('
Paragraph 1

Paragraph 2

Paragraph 3

5
4

', 'html-fragment');
$dom->filter('p')->last()->each(function($element) {
	$element->replace('hi');
});
echo $dom;
``

Expected:

Paragraph 1 Paragraph 2hi54
```
Actual:

Paragraph 1

Paragraph 2

Paragraph 3
hi

filter() reduces the current selection. You're filtering the current selection for any node that is a 'p' or has a 'p' descendant. The div has a p descendant so it matches. The div is replaced.

But the result is different from the jquery version as I posted above.
The reason why I don't use find because I can't find the root elements.
code:
$dom = FluentDOM::QueryCss('
Paragraph 1

Paragraph 2

Paragraph 3

5
4

', 'html-fragment');
$dom->find('p')->first()->replaceWith('hi');
echo $dom;

Result:

Paragraph 1

Paragraph 2

Paragraph 3

5hi

the first it finds is the p inside the div.

Hi, any updates?

The actual problem left seems to be with the CSS Selector to Xpath conversion. I added some tests using specific Xpath expressions and it works:
https://github.com/FluentDOM/FluentDOM/commit/a8c925b49933e88810bf6fb3d0e97005480c1767
This is difficult to debug because the conversion is not part of FluentDOM itself and an edge case.
I will look into it further but it will take time.


helm_resource: version

adds ability to provide --version argument to helm_resource

@nicks perhaps, but something more declarative is easier to consume and understand the behavior of; hence me not knowing of this way to provide this argument

I would be fine with a note on the readme on how to set --version. there's already a note about --set where you can put it.
I don't think we want to replicate every helm install flag as starlark arguments

i added some notes to the README and test on how to use --version
https://github.com/tilt-dev/tilt-extensions/tree/master/helm_resource#helm_resource


Added support for Organization Policy and Access Policy from CAI to Forseti inventory

This PR adds support for following policies from CAI.
Organization Policy

[x] cloudresourcemanager.googleapis.com/Organization
[x] cloudresourcemanager.googleapis.com/Folder
[x] cloudresourcemanager.googleapis.com/Project

Access Policy

[x] cloudresourcemanager.googleapis.com/Organization

@joecheuk will do the review here, in conjunction with the VPC-SC rego work.


Option for creating a task on the issue comment

Linked to issue #7.
New feature to create automatically a task for all violations which reach a severity threshold.

@mathieusimon, @nvillard
All pull-request comments has been integrated.

Ok for me!

Fine with me


Correct Tooltip Alignment

The tooltips in the Footer component are not aligned with their respective buttons and look a little odd:

I corrected the location of the Tooltip:

This was only a 3 line change, I adjusted left to %50 and translated along the X axis -50%, this should ensure it works across all screens. I also moved the tooltip closer to the button to prevent it fading away if you try to click the link inside the tooltip. This change is to the Footer's CSS.


TextInput: Remove PropTypes, NativeMethodsMixin; Convert to ES6 class

This pull requests converts TextInput to an ES6 class, and in the process removes its usage of prop-types and NativeMethodsMixin.
The code (and some relevant types) for the native components have been moved to TextInputNativeComponent.js.
The rest of the flow proptypes have been moved to TextInputTypes.js.
Test Plan:
These have both been tried in RNTester, and no behaviour regression has been observed.
Release Notes:
Help reviewers and the release process by writing your own release notes. See below for an example.
[GENERAL] [ENHANCEMENT] [Libraries/Components/TextInput/TextInput.js] - Removed Mixin and PropTypes, converted to ES6 class
[GENERAL] [ENHANCEMENT] [Libraries/Components/TextInput/TextInputNativeComponent.js] - Moved native component importing logic to its own file
[GENERAL] [ENHANCEMENT] [Libraries/Components/TextInput/TextInputTypes.js] - Moved Flow types to their own file

I'm currently stuck on some internal Jest tests which are triggering the TextInput's _onChange function via:
const {root} = ReactTestRenderer.create(<MyComponent />);
const instance = root.findByProps({ testID: 'my_test_id' });
const input = instance.findByType(TextInput);
input.instance._onChange({nativeEvent: {text}});

Jest's setup file is calling:
jest.mock('TextInput', () => mockComponent('TextInput'))

When migrating to forwardRef, TextInput is now the forwardRef function and not the underlying class so the mock isn't valid.
I was thinking that maybe the easiest thing to do is just move the class into its own file like TextInputInternal.js and change to mocking that file but it seems like a lot of leaking internals. Does anyone have an idea how to work around this?

@TheSavior I might be misunderstanding, but is it not finding the TextInput because it now uses forwardRef? If so, maybe this could be fixed by adding the displayName like https://github.com/facebook/react-native/pull/21950 did for View and Text?

Whoops, I should have responded here that I found a way to get past this but then I got stuck on something else and haven’t been able to look at this for the last few days. Haha.
I should be able to get back to this early next week or @rsnara can help get this one through internally in the mean time.


[CSL-1994] Move binaryTests to CborSpec

Also change the base branch, user stories must have separate branches, not develop.

Imo, it should apply only to features, maybe bugfixes, but not refactorings. I talked about it many times, but didn't get a proper response. I will bring it up again.

I ran tests locally and I see that serialization tests for the following types are too slow:

MsgHeaders, CommitmentsMap, DataMsg MCShares, USUndo

So you should decrease max success for them or use SmallGenerator wrapper. I don't mind leaving maxSuccess = 1000 for other binary tests (they are relatively fast).

I have realized that this goes directly against Duncan's suggestion:

actual tests should live with the package they are testing

Reverting.


LDAP push notification: can not find user by username

If LDAP is enabled, a push notification is handled by Portus and got error as follow:
  User Load (1.0ms)  SELECT  `users`.* FROM `users` WHERE `users`.`username` =<EMAIL_ADDRESS>LIMIT 1
  Cannot find user<EMAIL_ADDRESS>
To fix it, the code should be changed to use "ldap_name" to find user instead of "username" field.

A pull request with the following code was created.
  # Find the user that can be guessed from the given push event.
  def self.find_from_event(event)
    if Portus::LDAP.enabled?
      actor = User.find_by(ldap_name: event["actor"]["name"])
    else
      actor = User.find_by(username: event["actor"]["name"])
    end
    logger.error "Cannot find user #{event["actor"]["name"]}" if actor.nil?
    actor
  end

Fixed with PR #431.


Using libedit instead of readline

I noticed that the mariadb101u-server-10.1.12-1.ius.centos7.x86_64 packages use libedit for the client (like Fedora), instead of readline (like the IUS 10.0 packages and the stock MariaDB packages on centos 7)
I wonder if you would consider reverting this Fedora upstream change, or if other IUS users have opinions.
There may be license issues I am not fully aware of but I do see that Percona Server recently transitioned back to readline in client 5.6.24

Thanks @mikegriffin for bringing this up.
Generally we try to align with Fedora when building our packages (we even fork our spec files from Fedora spec files).  However, IUS by nature has to diverge from Fedora in order to meet our goals of building for RHEL and to convert the package into safe replacement packages.  This could be a case where it makes sense to use the RHEL approach (readline) instead of the Fedora approach (libedit).
More background on the Fedora change and the possible license issues can be found here.  I'm going to research this a bit more and allow some time for other users to provide feedback here.

Another consideration is that using libedit 3.1 on Fedora may make sense for their packages, but in EL7 we are limited to libedit 3.0.

https://bugzilla.redhat.com/show_bug.cgi?id=1079637
https://bugzilla.redhat.com/show_bug.cgi?id=1201988
https://bugzilla.redhat.com/show_bug.cgi?id=1187469
https://bugzilla.redhat.com/show_bug.cgi?id=1180403
https://bugzilla.redhat.com/show_bug.cgi?id=1202022
https://mariadb.atlassian.net/browse/MDEV-7777

A package has multiple binaries, some of them are GPLv2, some are GPLv3, and some are MIT licensed. In this case, you do need to list all of the individual licenses of the compiled binaries in the License tag, so it should read: License: GPLv2 and GPLv3 and MIT

https://fedoraproject.org/wiki/Licensing:FAQ?rd=Licensing/FAQ#How_should_I_handle_multiple_licensing_situations.3F
Based on that, licensing doesn't stop us from making this change.

Hey @mikegriffin,
I got some updated packages built with readline support.  Those packages will be hitting the testing repositories after tonight's automation run.  Let us know if they have the features you are looking for.

Hey @mikegriffin,
I am going to delay pushing the test packages with readline support, so we can get a fix out for #2.   We will get these packages into the testing repositories early next week.

Now that #2 has been address, mariadb101u 10.1.13-3 packages will be hitting the testing repositories after tonight's automation run.    Let us know if they have the features you are looking for.

@mikegriffin reached out over other means and confirmed this is the functionally he was looking for.   We will work on getting some 1:10.1.14-2.ius packages built with this feature.

Packages with this feature have been released.


No knowledge to solve this task based on previous tasks learned.

Noobies can't solve this task as in previous tasks there was no Functions learned. It is confusing for them. Before solving this task, need to introduce user with function construction etc. See all the previous tasks. Only variables, datatypes, operators. No functions, no methods.
Challenge Word Blanks has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

function wordBlanks(myNoun, myAdjective, myVerb, myAdverb) {
  var result = "";
  // Your code below this line

  // Your code above this line
  return result;
}

// Change the words here to test your function
wordBlanks("dog", "big", "ran", "quickly");

Consider these challenges as the preparation:
Concatenating Strings with Plus Operator Complete
Concatenating Strings with the Plus Equals Operator Complete
Constructing Strings with Variables Complete
Appending Variables to Strings
Ask for help in the chat room if you are still unsure of how to proceed.

I completely agree. The explanations do not provide enough concepts. only syntax to copy.
In addition to no explanation about functions, other concepts critical to this exercise that isn't explained:

passing arguments
return statement

I found the instructions for this exercise very confusing too. For simplicity's sake by not going into a lengthier explanation of functions, to proceed with this exercise, ignore the function construct.
Explaining the "return" statement also requires explaining functions, so all you need to know here is that the "result" variable of this return statement is the objective.
The var result = ""; line already declares the "result" variable name, so the point of the exercise is to build a single "result" String variable with the 4 variable names shown in the function line:
myNoun, myAdjective, myVerb, myAdverb
In other words, create a sentence using the 4 function variable names and assign that sentence to the "result" variable.
If you want to start learning how functions work, then follow the last step of the exercise's instructions by changing some one of the string words in the last line in the program.


🐛 App Check token generation returns error unknown..

Bug report
Describe the bug
When attempting phone authentication, I consistently encounter the captcha webview. Upon further investigation, I explored the Firebase App Check plugin. However, when attempting to obtain a token, the resulting token is encoded as eyJlcnJvciI6IlVOS05PV05fRVJST1IifQ==, and upon decoding, it reveals an unknown error. I suspect that the getToken result may be related to my continued encounter with the captcha webview.
Token content:
Click To Expand
{
  "error":"UNKNOWN_ERROR"
}

My Dart code:
Click To Expand
Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp(
    options: DefaultFirebaseOptions.currentPlatform,
  );
  await FirebaseAppCheck.instance.activate(
    androidProvider: AndroidProvider.playIntegrity,
  );
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: const MyHomePage(
        title: 'Flutter Demo Home Page',
      ),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({
    super.key,
    required this.title,
  });

  final String title;

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  String? token;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            ElevatedButton(
              onPressed: () async {
                token = await FirebaseAppCheck.instance.getToken();
                setState(() {});
              },
              child: const Text('Get AppCheck token'),
            ),
          ],
        ),
      ),
    );
  }
}

Steps to reproduce
Steps to reproduce the behavior:

Create a Firebase project
Add your SHA-256
Configure app check
Install the following packages:
firebase_auth: ^4.4.2
firebase_core: ^2.10.0
firebase_app_check: ^0.1.2+2
Configure your project with flutterfire configure
Run the code I've attached above

Expected behavior
To get a valid token.
Sample project
https://github.com/jopmiddelkamp/firebase_app_check_issue

Additional context

Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.7.12, on macOS 13.3.1 22E261 darwin-arm64, locale en-GB)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
[✓] Android Studio (version 2022.1)
[✓] Android Studio (version 2022.1)
[✓] VS Code (version 1.77.3)
[✓] Connected device (2 available)
[✓] HTTP Host Availability

• No issues found!

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
Dart SDK 2.19.6
Flutter SDK 3.7.12
firebase_auth_poc 1.0.0+1

dependencies:
- cupertino_icons 1.0.5
- dio 5.1.1 [http_parser meta path]
- firebase_app_check 0.1.2+2 [firebase_app_check_platform_interface firebase_app_check_web firebase_core firebase_core_platform_interface flutter]
- firebase_auth 4.4.2 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta]
- firebase_core 2.10.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- flutter 0.0.0 [characters collection js material_color_utilities meta vector_math sky_engine]
- freezed_annotation 2.2.0 [collection json_annotation meta]

dev dependencies:
- flutter_lints 2.0.1 [lints]
- flutter_test 0.0.0 [flutter test_api path fake_async clock stack_trace vector_math async boolean_selector characters collection js matcher material_color_utilities meta source_span stream_channel string_scanner term_glyph]
- freezed 2.3.2 [analyzer build build_config collection meta source_gen freezed_annotation json_annotation]

transitive dependencies:
- _fe_analyzer_shared 59.0.0 [meta]
- _flutterfire_internals 1.1.1 [collection firebase_core firebase_core_platform_interface flutter meta]
- analyzer 5.11.1 [_fe_analyzer_shared collection convert crypto glob meta package_config path pub_semver source_span watcher yaml]
- args 2.4.0
- async 2.10.0 [collection meta]
- boolean_selector 2.1.1 [source_span string_scanner]
- build 2.3.1 [analyzer async convert crypto glob logging meta path]
- build_config 1.1.1 [checked_yaml json_annotation path pubspec_parse yaml]
- characters 1.2.1
- checked_yaml 2.0.2 [json_annotation source_span yaml]
- clock 1.1.1
- collection 1.17.0
- convert 3.1.1 [typed_data]
- crypto 3.0.2 [typed_data]
- dart_style 2.3.0 [analyzer args path pub_semver source_span]
- fake_async 1.3.1 [clock collection]
- file 6.1.4 [meta path]
- firebase_app_check_platform_interface 0.0.6+1 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface]
- firebase_app_check_web 0.0.8+1 [_flutterfire_internals firebase_app_check_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins js]
- firebase_auth_platform_interface 6.13.1 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_auth_web 5.3.2 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser intl js meta]
- firebase_core_platform_interface 4.6.0 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.3.0 [firebase_core_platform_interface flutter flutter_web_plugins js meta]
- flutter_web_plugins 0.0.0 [flutter js characters collection material_color_utilities meta vector_math]
- glob 2.1.1 [async collection file path string_scanner]
- http_parser 4.0.2 [collection source_span string_scanner typed_data]
- intl 0.17.0 [clock path]
- js 0.6.5 [meta]
- json_annotation 4.8.0 [meta]
- lints 2.0.1
- logging 1.1.1
- matcher 0.12.13 [meta stack_trace]
- material_color_utilities 0.2.0
- meta 1.8.0
- package_config 2.1.0 [path]
- path 1.8.2
- plugin_platform_interface 2.1.4 [meta]
- pub_semver 2.1.3 [collection meta]
- pubspec_parse 1.2.2 [checked_yaml collection json_annotation pub_semver yaml]
- sky_engine 0.0.99
- source_gen 1.2.7 [analyzer async build dart_style glob path source_span yaml]
- source_span 1.9.1 [collection path term_glyph]
- stack_trace 1.11.0 [path]
- stream_channel 2.1.1 [async]
- string_scanner 1.2.0 [source_span]
- term_glyph 1.2.1
- test_api 0.4.16 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph matcher]
- typed_data 1.3.1 [collection]
- vector_math 2.1.4
- watcher 1.0.2 [async path]
- yaml 3.1.1 [collection source_span string_scanner]

Hi @jopmiddelkamp, what platform are you experiencing this issue on? Can you also share the code that is used to verify the token? I see that you make a network request to verify the token but the URL is localhost.
Thank you

I'm experiencing this on Android.
Here is the code I use for validation.
// Import the necessary packages
const express = require('express');
const app = express();
const admin = require('firebase-admin');
var serviceAccount = require("./firebase-adminsdk.json");

// Initialize the Firebase app
const firebaseApp = admin.initializeApp({
  credential: admin.credential.cert(serviceAccount)
});

// Middleware function to verify the App Check token
const appCheckVerification = async (req, res, next) => {
    const appCheckToken = req.header('X-Firebase-AppCheck');
    console.log('App Check token: \'' + appCheckToken + '\'');

    if (!appCheckToken) {
        res.status(401);
        return next('Unauthorized');
    }

    try {
        const appCheckClaims = await admin.appCheck().verifyToken(appCheckToken);
        console.log(appCheckClaims);

        // If verifyToken() succeeds, continue with the next middleware
        // function in the stack.
        return next();
    } catch (err) {
        console.log(err);
        res.status(401);
        return next('Unauthorized');
    }
}

// Define your API endpoint and use the middleware function
app.get('/check-token', [appCheckVerification], (req, res) => {
    console.log('App Check token is valid');
});

// Start the Express app
app.listen(3000, () => {
    console.log('App listening on port 3000!');
});

Just note that in this context it is irrelevant as the token that gets returned contains a json with a single error unkown field. There is no point in verifying it if the body is.
{
  "error":"UNKNOWN_ERROR"
}

I think I might not have set it up correctly. This is my log when I attempt to get the token.
@jopmiddelkamp, can you share your logs when you run the app and try to get the token?
Syncing files to device Pixel 7...
I/.example.app_ct(20803): Compiler allocated 4413KB to compile void android.view.ViewRootImpl.performTraversals()
E/SurfaceSyncer(20803): Failed to find sync for id=0
D/TrafficStats(20803): tagSocket(141) with statsTag=0xffffffff, statsUid=-1
I/PlayCore(20803): UID: [10288]  PID: [20803] IntegrityService : requestIntegrityToken(IntegrityTokenRequest{nonce=AaK4wXBWXZN88MO8Y31agLE98zf3-XzRpO--JfmdUuihKGR15Bf3NUq6YdyOdRysbbsCBfZiWeJeNkEm4PMgsiYFrjUbrlzvilXZs0G7Z95cg3_5uhkDClolMaPgdy4BVERtKeDG4JNh5UQCi6FBlmpVINTWfEa3gUVNkenAlgZK34E=, cloudProjectNumber=192368888880})
I/PlayCore(20803): UID: [10288]  PID: [20803] IntegrityService : Initiate binding to the service.
I/PlayCore(20803): UID: [10288]  PID: [20803] IntegrityService : ServiceConnectionImpl.onServiceConnected(ComponentInfo{com.android.vending/com.google.android.finsky.integrityservice.IntegrityService})
I/PlayCore(20803): UID: [10288]  PID: [20803] IntegrityService : linkToDeath
I/PlayCore(20803): UID: [10288]  PID: [20803] OnRequestIntegrityTokenCallback : onRequestIntegrityToken
I/PlayCore(20803): UID: [10288]  PID: [20803] IntegrityService : Unbind from service.
E/flutter (20803): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: [firebase_app_check/unknown] com.google.firebase.FirebaseException: Error returned from API. code: 403 body: App attestation failed.
E/flutter (20803): #0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:653:7)
E/flutter (20803): #1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:315:18)
E/flutter (20803): <asynchronous suspension>
E/flutter (20803): #2      MethodChannelFirebaseAppCheck.getToken (package:firebase_app_check_platform_interface/src/method_channel/method_channel_firebase_app_check.dart:100:22)
E/flutter (20803): <asynchronous suspension>
E/flutter (20803): #3      _MyHomePageState.build.<anonymous closure> (package:app_ct/main.dart:62:25)
E/flutter (20803): <asynchronous suspension>
E/flutter (20803):
I/.example.app_ct(20803): Background concurrent copying GC freed 43660(4147KB) AllocSpace objects, 13(516KB) LOS objects, 90% free, 2729KB/26MB, paused 223us,115us total 102.927ms
W/JavaBinder(20803): BinderProxy is being destroyed but the application did not call unlinkToDeath to unlink all of its death recipients beforehand.  Releasing leaked death recipient: com.google.android.play.integrity.internal.zzk
I/BpBinder(20803): onLastStrongRef automatically unlinking death recipients: <uncached descriptor>

I'm a bit busy at the moment. I'll reply as soon as possible.

@danagbemava-nc
Sorry for this very late reply.. Some things came in between and I lost track of this issue. This is what I see when I press the get token button.
Launching lib/main.dart on SM G973F in debug mode...
main.dart:1
✓  Built build/app/outputs/flutter-apk/app-debug.apk.
Connecting to VM Service at ws://<IP_ADDRESS>:62930/bBmK00ihH4A=/ws
I/Gralloc4( 3151): mapper 4.x is not supported
W/Gralloc3( 3151): mapper 3.x is not supported
I/gralloc ( 3151): Arm Module v1.0
W/Gralloc4( 3151): allocator 4.x is not supported
W/Gralloc3( 3151): allocator 3.x is not supported
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Creating frameDrawingCallback nextDrawUseBlastSync=false reportNextDraw=true hasBlurUpdates=false
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Creating frameCompleteCallback
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Received frameDrawingCallback frameNum=1. Creating transactionCompleteCallback=false
I/SurfaceView@4bcc918( 3151): uSP: rtp = Rect(0, 0 - 1080, 2241) rtsw = 1080 rtsh = 2241
I/SurfaceView@4bcc918( 3151): onSSPAndSRT: pl = 0 pt = 0 sx = 1.0 sy = 1.0
I/SurfaceView@4bcc918( 3151): aOrMT: uB = true t = android.view.SurfaceControl$Transaction@36c0e45 fN = 1 android.view.SurfaceView.access$500:124 android.view.SurfaceView$SurfaceViewPositionUpdateListener.positionChanged:1728 android.graphics.RenderNode$CompositePositionUpdateListener.positionChanged:319
I/SurfaceView@4bcc918( 3151): aOrMT: vR.mWNT, vR = ViewRootImpl@6294ba9[MainActivity]
I/ViewRootImpl@6294ba9[MainActivity]( 3151): mWNT: t = android.view.SurfaceControl$Transaction@36c0e45 fN = 1 android.view.SurfaceView.applyOrMergeTransaction:1628 android.view.SurfaceView.access$500:124 android.view.SurfaceView$SurfaceViewPositionUpdateListener.positionChanged:1728
I/ViewRootImpl@6294ba9[MainActivity]( 3151): mWNT: merge t to BBQ
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Received frameCompleteCallback  lastAcquiredFrameNum=1 lastAttemptedDrawFrameNum=1
I/ViewRootImpl@6294ba9[MainActivity]( 3151): [DP] pdf(0) 1 android.view.ViewRootImpl.lambda$addFrameCompleteCallbackIfNeeded$3$ViewRootImpl:5000 android.view.ViewRootImpl$$ExternalSyntheticLambda16.run:6 android.os.Handler.handleCallback:938
I/ViewRootImpl@6294ba9[MainActivity]( 3151): [DP] rdf()
D/ViewRootImpl@6294ba9[MainActivity]( 3151): reportDrawFinished (fn: -1)
I/ViewRootImpl@6294ba9[MainActivity]( 3151): MSG_WINDOW_FOCUS_CHANGED 1 1
D/InputMethodManager( 3151): startInputInner - Id : 0
I/InputMethodManager( 3151): startInputInner - mService.startInputOrWindowGainedFocus
D/InputMethodManager( 3151): startInputInner - Id : 0
I/ViewRootImpl@6294ba9[MainActivity]( 3151): ViewPostIme pointer 0
I/ViewRootImpl@6294ba9[MainActivity]( 3151): ViewPostIme pointer 1
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : requestIntegrityToken(IntegrityTokenRequest{nonce=AaK4wXDXhYBFbCDgj-dmB7NGmUCfqG7crs8A8NNUR1APmj5W-hD9U8ZgWvW72tpx6gt2UdPVLbMUuCirBKMT7EKQtbQ7NB4u-G_4JjqbDT4RSO-L-Ovfz507DA45gVbhKefeKqp9ZIG6vTt0FbQ5j1i3NB-zAdA_HgkwlAiTo1N0bw==, cloudProjectNumber=859123377401})
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : Initiate binding to the service.
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : ServiceConnectionImpl.onServiceConnected(ComponentInfo{com.android.vending/com.google.android.finsky.integrityservice.IntegrityService})
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : linkToDeath
I/PlayCore( 3151): UID: [10604]  PID: [3151] OnRequestIntegrityTokenCallback : onRequestIntegrityToken
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : Unbind from service.

Could you also reopen this @danagbemava-nc?

@danagbemava-nc  I only have our production app available in the play store. We do have a staging app setup in the play console but it is internal test only. Is that also enough?
How do you normally set this up? I would like to test it before pushing anything to production.

Hi @jopmiddelkamp, I'm not entirely sure. I'm not too familiar with app_check.
Can you check https://github.com/firebase/flutterfire/issues/10593 to see if any of the comments in there help you out?

@danagbemava-nc I only have our production app available in the play store. We do have a staging app setup in the play console but it is internal test only. Is that also enough?
How do you normally set this up? I would like to test it before pushing anything to production.

That's the same issue i am facing now. so if you fix the issue. please tell me how to do it?

@danagbemava-nc can you not ask some other people (maybe even the people that created the service) from the Google team how they intended this to be used?

Labeling for further insight from the team.
/cc @russellwheatley

any update on this issue ?

Would also love to know a status update.

@danagbemava-nc
Do you know if Firebase Remote Config, just as Firestore, uses Firebase App Check as part of its security designed to protect its access from unauthorised or abusive requests?
Otherwise I might use Firebase Remote Config te set an API key and store this key in the phone its memory and use this key in the HTTP headers on my HTTPS endpoint requests to verify if the request is coming from a real device. But it only works if Firebase Remote Config is using Firebase App Check as part of its security. What are your thoughts on this?

@jopmiddelkamp, Firebase Remote config doesn't seem to use app check. See https://firebase.google.com/docs/app-check.

After days spent to just make it work. I followed all documentation and it works with success when using de debug token. But in production, no way. All installed all certeficates SHA-256, SHA-1 (play store and sign deploy certifcates) both in firebase app settings and in appcheck... But no way, i got 403 error.
As a good solution was: Remove Appcheck from the project and rollback the project and implement my custom verication app. I took just one day to do that rather than using Google AppCheck that never will work in production.
I hope that it works for you. For me it's a just a big useless api that makes you losing your time and your money for something will never work.

@skonsoftSASU  wat is your previous solution if you mind me asking?

@jopmiddelkamp I ended by using a package_info_plus (buildSignature) to generate a runtime signature and for critical operation, i join the signature to request which compared to the reference signature stored in the server. If they did not match, the app was cracked.
This is valid only for android. I will check how to implement same logic for ios.
This solution is more simple and secures the APP... it took 2 hours of work rather than a couple of days for appcheck without any success.

@danagbemava-nc
Sorry for this very late reply.. Some things came in between and I lost track of this issue. This is what I see when I press the get token button.
Launching lib/main.dart on SM G973F in debug mode...
main.dart:1
✓  Built build/app/outputs/flutter-apk/app-debug.apk.
Connecting to VM Service at ws://<IP_ADDRESS>:62930/bBmK00ihH4A=/ws
I/Gralloc4( 3151): mapper 4.x is not supported
W/Gralloc3( 3151): mapper 3.x is not supported
I/gralloc ( 3151): Arm Module v1.0
W/Gralloc4( 3151): allocator 4.x is not supported
W/Gralloc3( 3151): allocator 3.x is not supported
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Creating frameDrawingCallback nextDrawUseBlastSync=false reportNextDraw=true hasBlurUpdates=false
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Creating frameCompleteCallback
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Received frameDrawingCallback frameNum=1. Creating transactionCompleteCallback=false
I/SurfaceView@4bcc918( 3151): uSP: rtp = Rect(0, 0 - 1080, 2241) rtsw = 1080 rtsh = 2241
I/SurfaceView@4bcc918( 3151): onSSPAndSRT: pl = 0 pt = 0 sx = 1.0 sy = 1.0
I/SurfaceView@4bcc918( 3151): aOrMT: uB = true t = android.view.SurfaceControl$Transaction@36c0e45 fN = 1 android.view.SurfaceView.access$500:124 android.view.SurfaceView$SurfaceViewPositionUpdateListener.positionChanged:1728 android.graphics.RenderNode$CompositePositionUpdateListener.positionChanged:319
I/SurfaceView@4bcc918( 3151): aOrMT: vR.mWNT, vR = ViewRootImpl@6294ba9[MainActivity]
I/ViewRootImpl@6294ba9[MainActivity]( 3151): mWNT: t = android.view.SurfaceControl$Transaction@36c0e45 fN = 1 android.view.SurfaceView.applyOrMergeTransaction:1628 android.view.SurfaceView.access$500:124 android.view.SurfaceView$SurfaceViewPositionUpdateListener.positionChanged:1728
I/ViewRootImpl@6294ba9[MainActivity]( 3151): mWNT: merge t to BBQ
D/ViewRootImpl@6294ba9[MainActivity]( 3151): Received frameCompleteCallback  lastAcquiredFrameNum=1 lastAttemptedDrawFrameNum=1
I/ViewRootImpl@6294ba9[MainActivity]( 3151): [DP] pdf(0) 1 android.view.ViewRootImpl.lambda$addFrameCompleteCallbackIfNeeded$3$ViewRootImpl:5000 android.view.ViewRootImpl$$ExternalSyntheticLambda16.run:6 android.os.Handler.handleCallback:938
I/ViewRootImpl@6294ba9[MainActivity]( 3151): [DP] rdf()
D/ViewRootImpl@6294ba9[MainActivity]( 3151): reportDrawFinished (fn: -1)
I/ViewRootImpl@6294ba9[MainActivity]( 3151): MSG_WINDOW_FOCUS_CHANGED 1 1
D/InputMethodManager( 3151): startInputInner - Id : 0
I/InputMethodManager( 3151): startInputInner - mService.startInputOrWindowGainedFocus
D/InputMethodManager( 3151): startInputInner - Id : 0
I/ViewRootImpl@6294ba9[MainActivity]( 3151): ViewPostIme pointer 0
I/ViewRootImpl@6294ba9[MainActivity]( 3151): ViewPostIme pointer 1
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : requestIntegrityToken(IntegrityTokenRequest{nonce=AaK4wXDXhYBFbCDgj-dmB7NGmUCfqG7crs8A8NNUR1APmj5W-hD9U8ZgWvW72tpx6gt2UdPVLbMUuCirBKMT7EKQtbQ7NB4u-G_4JjqbDT4RSO-L-Ovfz507DA45gVbhKefeKqp9ZIG6vTt0FbQ5j1i3NB-zAdA_HgkwlAiTo1N0bw==, cloudProjectNumber=859123377401})
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : Initiate binding to the service.
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : ServiceConnectionImpl.onServiceConnected(ComponentInfo{com.android.vending/com.google.android.finsky.integrityservice.IntegrityService})
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : linkToDeath
I/PlayCore( 3151): UID: [10604]  PID: [3151] OnRequestIntegrityTokenCallback : onRequestIntegrityToken
I/PlayCore( 3151): UID: [10604]  PID: [3151] IntegrityService : Unbind from service.

There is no exception in these logs. We already have an issue open for exceptions that occur when calling getToken() when the Play Integrity provider is used. I'm going to close this out and move the discussion there.


Updated the list of pre-processors

Added a larger variety of pre-processors

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

@takachou Thanks for this pull request (PR).
Unfortunately, we are marking this PR invalid for hacktoberfest. We believe it does not follow the recommended guidelines:

Quality Standards
In line with Hacktoberfest value #2 (Quantity is fun, Quality is key), we have provided examples of the quality >standards we encourage. This applies mainly to beginners.

PRs that are automated e.g. scripted opening PRs to remove whitespace / optimize images.
PRs that are disruptive e.g. taking someone else's branch/commits and making a PR.
PRs that are regarded by a project maintainer as a hindrance vs. helping.
Something that's clearly an attempt to simply +1 your PR count for October.

Learn more at https://hacktoberfest.digitalocean.com/details


When removing multiple blocks, Ctrl+Z doesn't return all of them

Description
When removing multiple blocks, Ctrl+Z doesn't return all of them
Environment

Editor.js version: 2.29.1
editorjs-undo version: 2.0.28
Device, Browser, OS: Desktop, Windows 10, Google Chrome 128.0.6613.139

Current behavior

Select few paragraphs with mouse
Press 'Backspace' to delete them
Press 'Ctrl+Z'

-- Paragraphs doesn't return. (Sometimes, only first of them returns)
-- Console shows warning like 'There is no block at index 1684'
Expected behavior
Paragraphs return

Same issue, but when trying to delete it from tool one by one, it works(undo/redo)

Same issue, any updates?

I am also facing issues with Undo and Redo when selected multiple blocks it's not working. Any help would be appreciated.

If you use the stable version 1.x.x, it works. I reverted to that one


make model/collection parsing an option in parseModelAndCollection

When parseModelAndCollection is called by the rendr-handlebars view method on the client-side there is not an app option present, so if the models parse method uses this.app it will error. Beyond that it seems unnecessary to call parse in this use-case which is just constructing the fetch summary for the placeholder div.
We do want the model/collection to be instantiated using parse when the view is attached, so pass { parse: true } explicitly from parseOptions
@saponifi3d @mattpardee @mrubens

Coverage remained the same when pulling 902e5ce11e4a99392a0de45cb7e3f157bd20fdd7 on change:md_option_parse into 1eddc2618b76c46d875837161b27d8a14f2e0fe2 on rendrjs:master.

Coverage increased (+0.08%) when pulling 4b6ca85e75618c13f999f74afdeb62d8192f622d on change:md_option_parse into 1eddc2618b76c46d875837161b27d8a14f2e0fe2 on rendrjs:master.

Coverage remained the same when pulling 9ee3a423b7350282c59725de8e114222994bb7e1 on change:md_option_parse into 43623911db92be60379558c263b13183607f1edc on rendrjs:master.

Coverage remained the same when pulling 020e0faa598af489743a173857bc4a8cfac28579 on change:md_option_parse into 43623911db92be60379558c263b13183607f1edc on rendrjs:master.


Invalid address length, expected 40 got 34 bytes

JsonRpcError: Invalid address length, expected 40 got 34 bytes
    at JsonRpcError.ExtendableBuiltin (file:///C:/Users/prem2/AppData/Local/Programs/MintMeCoinWallet/resources/app.asar/app/index.js:25716:30)
    at new JsonRpcError (file:///C:/Users/prem2/AppData/Local/Programs/MintMeCoinWallet/resources/app.asar/app/index.js:25745:116)
    at file:///C:/Users/prem2/AppData/Local/Programs/MintMeCoinWallet/resources/app.asar/app/index.js:25782:19

JsonRpcError: Invalid address length, expected 40 got 34 bytes
    at JsonRpcError.ExtendableBuiltin (file:///C:/Users/prem2/AppData/Local/Programs/MintMeCoinWallet/resources/app.asar/app/index.js:25716:30)
    at new JsonRpcError (file:///C:/Users/prem2/AppData/Local/Programs/MintMeCoinWallet/resources/app.asar/app/index.js:25745:116)
    at file:///C:/Users/prem2/AppData/Local/Programs/MintMeCoinWallet/resources/app.asar/app/index.js:25782:19

@CodePrem  Please recheck your address and make sure it's the right one.


is API access with JWT tokens working yet?

Hey acbuyan,
I have been silent for a while as our project got interrupted for a few months, but I am now back at working on automized download of the BioCollect data for a specific project, including all fields available as well as the media (photos of the Koala).
First I generated a list with all field names (n=760) to use these rather than the "basic" setting for downloading atlas occurences of my research UID.
After receiving error "Expecting value: line 1 column 1 (char0)" a JSON decode error in the request call in atlas_occurrences.py, I added a line to print the raw server response in that python module.
That showed me that querying all 760 fields produced an URL too long to handle (caused an error 414).
I therefore wrote a loop around this to download the fields in chunks and stitch the data back together. This however then caused a new error (now status: in Queue), error thrown because of line:
l 311: return pd.read_csv(zipfile.ZipFile(io.BytesIO(zipURL.content)).open('data.csv'),low_memory=False)
apparently there was a pandas data error "No columns to parse from file".
With this I also received an automatic email from ALA Support that the "Occurrence download failed - data". I did reply to that but haven't heard back yet.
When I read through the ALA API doc, I found that the API I (likely) need to access all data occurrences of that project may be protected?: /occurrences/{recordUuid} (retrieve full record details).
I am unsure if this is the correct API though.
Yet it seems like the JWT token access to protected API is included but not yet functional in galah-python? Judging from the galah_config.py script and the generate_jwt_token.py script?
I have already applied and received a ClientID and Secret and would love to see if this allows me to get the required data of the project...
Yet you have done such incredible work with this package already, that I would rather try and inquire how far JWT tokens are implemented here yet, and if this is the way to solve my access and download issues reaching far beyond the "basic" field list.
I am not a software developer or have experience with APIs, I am more a data analyst. In this case however I can't reach the data I need to analyse, haha, so if there is any way I could try to help with this issue/further development, please let me know.
Hope you can point me in the correct direction, happy to give more specific project details if needed.
Thanks again!
Jojo

Hi Jojo,
Good to hear from you, and great to hear that you continue to use galah-python!  It makes my day :)
First things first: there have been a couple of updates to galah-python; have you updated to the latest version? I don't think this will solve your issue, but I have fixed some bugs along the way and added other options too.
As far as your question on JWT tokens, we’re holding off on JWT token integration until we deploy Cognito. That should be soon but we’re not sure.  In addition, JWT tokens are only for access to sensitive data, and only then to access that you have personally been approved to receive. If these conditions do not apply, then JWT tokens are not what's causing your problem.
Unfortunately, galah-python doesn’t support BioCollect APIs, so you can only use galah to get occurrences once they have been passed to biocache.
I may have asked you this before, but what is the reason that you are downloading all fields?  It is unfortunately never a good idea to download all of the ALA fields, as they include spatial data that cannot be relevant to every application. It also includes Darwin Core fields that are rarely populated and may not be useful to you. If you know what data fields you need, you can then include it as an argument to atlas_occurrences

Hey acbuyan,
Haha, you're welcome! I'm very glad this package exists!
It could very well be that I am looking for the issues in the wrong place 😕 I think JWT tokens were recommended to me by the ALA support mid last year to access BioCollect APIs? Not sure.
But maybe I can work with the galah-python package afterall, the data shouldn't be sensitive. So from what you said it should work! Yes, I have updated the package 🙂
I'll keep playing around and see if I can finally get it downloaded, it shouldn't be this hard haha 😅.
Regarding the field names: I just found that all the field names had such weird names, that I couldn't find the ones I actually need.
I have solved that issue now by downloading the dataset manually once, and extracted the field names from the headings.csv file to use for the download (198 instead of all 760).
I don't know if you remember, but the overall project goal is: I basically want to download new occurrences in this dataset at a frequency xy.
These downloads are then converted to a suitable ESRI format to display it on a map. So the ESRI dashboard we are working on should be kept up-to-date without having to download the ALA KoalaWatch data manually all the time.
The last time I had downloaded the initial data was in July last year, new occurrences since then were 1155, according to the atlas_count function.
I am sorry for the long email below, maybe you can help. I still can't get data downloaded, the results from atlas_occurrences are empty.
No error messages, the console print with the queries are inserted below.
Despite reducing the fields to download to 198, I do not get any data:
When I try to download these 1155 new ones, I end up with an empty data frame. (Console prints are in Consolas font).
query as follows:
df_occurrences_new = galah.atlas_occurrences(filters=filter, verbose=True, fields=fields)
filter = ["taxa==Phascolarctos cinereus", "dataResourceUid==dr19064", "eventDate>=" + str(last_download_date)]
last_download_date:  2023-07-10T14:23:19Z
Below the console print (verbose in atlas_occurrences = True 🙂):
length of list of field names from headings.csv:  198
list of field names from headings.csv:  ['images', 'raw_recordedBy', 'dcterms:modified', 'dcterms:language', 'dcterms:license', 'rightsHolder', 'dcterms:accessRights', 'dcterms:bibliographicCitation', 'references', 'institutionID', 'collectionID', 'datasetID', 'institutionCode', 'collectionCode', 'datasetName', 'ownerInstitutionCode', 'basisOfRecord', 'informationWithheld', 'dataGeneralizations', 'dynamicProperties', 'occurrenceID', 'catalogNumber', 'recordNumber', 'recordedBy', 'individualCount', 'organismQuantity', 'organismQuantityType', 'sex', 'lifeStage', 'reproductiveCondition', 'behavior', 'establishmentMeans', 'occurrenceStatus', 'preparations', 'disposition', 'associatedMedia', 'associatedReferences', 'associatedSequences', 'associatedTaxa', 'otherCatalogNumbers', 'occurrenceRemarks', 'organismID', 'organismName', 'organismScope', 'associatedOccurrences', 'associatedOrganisms', 'previousIdentifications', 'organismRemarks', 'materialSampleID', 'eventID', 'parentEventID', 'fieldNumber', 'eventDate', 'eventTime', 'startDayOfYear', 'endDayOfYear', 'year', 'month', 'day', 'verbatimEventDate', 'habitat', 'samplingProtocol', 'samplingEffort', 'sampleSizeValue', 'sampleSizeUnit', 'fieldNotes', 'eventRemarks', 'locationID', 'higherGeographyID', 'higherGeography', 'continent', 'waterBody', 'islandGroup', 'island', 'country', 'countryCode', 'stateProvince', 'county', 'municipality', 'locality', 'verbatimLocality', 'minimumElevationInMeters', 'maximumElevationInMeters', 'verbatimElevation', 'minimumDepthInMeters', 'maximumDepthInMeters', 'verbatimDepth', 'minimumDistanceAboveSurfaceInMeters', 'maximumDistanceAboveSurfaceInMeters', 'locationAccordingTo', 'locationRemarks', 'decimalLatitude', 'decimalLongitude', 'geodeticDatum', 'coordinateUncertaintyInMeters', 'coordinatePrecision', 'pointRadiusSpatialFit', 'verbatimCoordinates', 'verbatimLatitude', 'verbatimLongitude', 'verbatimCoordinateSystem', 'verbatimSRS', 'footprintWKT', 'footprintSRS', 'footprintSpatialFit', 'georeferencedBy', 'georeferencedDate', 'georeferenceProtocol', 'georeferenceSources', 'georeferenceVerificationStatus', 'georeferenceRemarks', 'geologicalContextID', 'earliestEonOrLowestEonothem', 'latestEonOrHighestEonothem', 'earliestEraOrLowestErathem', 'latestEraOrHighestErathem', 'earliestPeriodOrLowestSystem', 'latestPeriodOrHighestSystem', 'earliestEpochOrLowestSeries', 'latestEpochOrHighestSeries', 'earliestAgeOrLowestStage', 'latestAgeOrHighestStage', 'lowestBiostratigraphicZone', 'highestBiostratigraphicZone', 'lithostratigraphicTerms', 'group', 'formation', 'member', 'bed', 'identificationID', 'identificationQualifier', 'typeStatus', 'identifiedBy', 'dateIdentified', 'identificationReferences', 'identificationVerificationStatus', 'identificationRemarks', 'taxonID', 'scientificNameID', 'acceptedNameUsageID', 'parentNameUsageID', 'originalNameUsageID', 'nameAccordingToID', 'namePublishedInID', 'taxonConceptID', 'raw_scientificName', 'scientificName', 'acceptedNameUsage', 'parentNameUsage', 'originalNameUsage', 'nameAccordingTo', 'namePublishedIn', 'namePublishedInYear', 'higherClassification', 'kingdom', 'phylum', 'class', 'order', 'family', 'genus', 'subgenus', 'specificEpithet', 'infraspecificEpithet', 'taxonRank', 'verbatimTaxonRank', 'scientificNameAuthorship', 'vernacularName', 'nomenclaturalCode', 'taxonomicStatus', 'nomenclaturalStatus', 'taxonRemarks', 'measurementDeterminedDate', 'videos', 'species', 'relationshipAccordingTo', 'resourceRelationshipID', 'measurementRemarks', 'measurementValue', 'relationshipRemarks', 'resourceID', 'relationshipEstablishedDate', 'identifierRole', 'relationshipOfResource', 'measurementMethod', 'identifier', 'provenance', 'sounds', 'recordID', 'rights', 'source', 'measurementID', 'measurementType', 'measurementUnit', 'measurementDeterminedBy', 'measurementAccuracy', 'relatedResourceID', 'images', 'dcterms:type']
downloading RCC data since last download.
last_download_date:  2023-07-10T14:23:19Z
koala_watch_occurrences_count:
totalRecords
0          1155
payload for queryID: {'fq': ['taxa:"Phascolarctos cinereus"', 'dataResourceUid:"dr19064"', 'eventDate:[2023-07-10T14:23:19Z TO *]']}
queryID URL: https://biocache-ws.ala.org.au/ws/webportal/params
method: POST
qid for query:<PHONE_NUMBER>220
URL for result:https://biocache-ws.ala.org.au/ws/occurrences/offline/download?email=jojo.schultz%40outlook.de&dwcHeaders=True&reasonTypeId=4&emailNotify=false&fq=(qid%3A1709770399220)&fields=images%2Craw_recordedBy%2Cdcterms:modified%2Cdcterms:language%2Cdcterms:license%2CrightsHolder%2Cdcterms:accessRights%2Cdcterms:bibliographicCitation%2Creferences%2CinstitutionID%2CcollectionID%2CdatasetID%2CinstitutionCode%2CcollectionCode%2CdatasetName%2CownerInstitutionCode%2CbasisOfRecord%2CinformationWithheld%2CdataGeneralizations%2CdynamicProperties%2CoccurrenceID%2CcatalogNumber%2CrecordNumber%2CrecordedBy%2CindividualCount%2CorganismQuantity%2CorganismQuantityType%2Csex%2ClifeStage%2CreproductiveCondition%2Cbehavior%2CestablishmentMeans%2CoccurrenceStatus%2Cpreparations%2Cdisposition%2CassociatedMedia%2CassociatedReferences%2CassociatedSequences%2CassociatedTaxa%2CotherCatalogNumbers%2CoccurrenceRemarks%2CorganismID%2CorganismName%2CorganismScope%2CassociatedOccurrences%2CassociatedOrganisms%2CpreviousIdentifications%2CorganismRemarks%2CmaterialSampleID%2CeventID%2CparentEventID%2CfieldNumber%2CeventDate%2CeventTime%2CstartDayOfYear%2CendDayOfYear%2Cyear%2Cmonth%2Cday%2CverbatimEventDate%2Chabitat%2CsamplingProtocol%2CsamplingEffort%2CsampleSizeValue%2CsampleSizeUnit%2CfieldNotes%2CeventRemarks%2ClocationID%2ChigherGeographyID%2ChigherGeography%2Ccontinent%2CwaterBody%2CislandGroup%2Cisland%2Ccountry%2CcountryCode%2CstateProvince%2Ccounty%2Cmunicipality%2Clocality%2CverbatimLocality%2CminimumElevationInMeters%2CmaximumElevationInMeters%2CverbatimElevation%2CminimumDepthInMeters%2CmaximumDepthInMeters%2CverbatimDepth%2CminimumDistanceAboveSurfaceInMeters%2CmaximumDistanceAboveSurfaceInMeters%2ClocationAccordingTo%2ClocationRemarks%2CdecimalLatitude%2CdecimalLongitude%2CgeodeticDatum%2CcoordinateUncertaintyInMeters%2CcoordinatePrecision%2CpointRadiusSpatialFit%2CverbatimCoordinates%2CverbatimLatitude%2CverbatimLongitude%2CverbatimCoordinateSystem%2CverbatimSRS%2CfootprintWKT%2CfootprintSRS%2CfootprintSpatialFit%2CgeoreferencedBy%2CgeoreferencedDate%2CgeoreferenceProtocol%2CgeoreferenceSources%2CgeoreferenceVerificationStatus%2CgeoreferenceRemarks%2CgeologicalContextID%2CearliestEonOrLowestEonothem%2ClatestEonOrHighestEonothem%2CearliestEraOrLowestErathem%2ClatestEraOrHighestErathem%2CearliestPeriodOrLowestSystem%2ClatestPeriodOrHighestSystem%2CearliestEpochOrLowestSeries%2ClatestEpochOrHighestSeries%2CearliestAgeOrLowestStage%2ClatestAgeOrHighestStage%2ClowestBiostratigraphicZone%2ChighestBiostratigraphicZone%2ClithostratigraphicTerms%2Cgroup%2Cformation%2Cmember%2Cbed%2CidentificationID%2CidentificationQualifier%2CtypeStatus%2CidentifiedBy%2CdateIdentified%2CidentificationReferences%2CidentificationVerificationStatus%2CidentificationRemarks%2CtaxonID%2CscientificNameID%2CacceptedNameUsageID%2CparentNameUsageID%2CoriginalNameUsageID%2CnameAccordingToID%2CnamePublishedInID%2CtaxonConceptID%2Craw_scientificName%2CscientificName%2CacceptedNameUsage%2CparentNameUsage%2CoriginalNameUsage%2CnameAccordingTo%2CnamePublishedIn%2CnamePublishedInYear%2ChigherClassification%2Ckingdom%2Cphylum%2Cclass%2Corder%2Cfamily%2Cgenus%2Csubgenus%2CspecificEpithet%2CinfraspecificEpithet%2CtaxonRank%2CverbatimTaxonRank%2CscientificNameAuthorship%2CvernacularName%2CnomenclaturalCode%2CtaxonomicStatus%2CnomenclaturalStatus%2CtaxonRemarks%2CmeasurementDeterminedDate%2Cvideos%2Cspecies%2CrelationshipAccordingTo%2CresourceRelationshipID%2CmeasurementRemarks%2CmeasurementValue%2CrelationshipRemarks%2CresourceID%2CrelationshipEstablishedDate%2CidentifierRole%2CrelationshipOfResource%2CmeasurementMethod%2Cidentifier%2Cprovenance%2Csounds%2CrecordID%2Crights%2Csource%2CmeasurementID%2CmeasurementType%2CmeasurementUnit%2CmeasurementDeterminedBy%2CmeasurementAccuracy%2CrelatedResourceID%2Cimages%2Cdcterms:type%2C&qa=none&flimit=-1
method: GET
Data for download:
https://biocache.ala.org.au/biocache-download/7d58aab5-1e0f-384d-ba9a-f632fb210b51/1709770399748/data.zip
size of dataframe (new data entries):  0
I also tried to download all data again (totalRecords n=2696), without passing an eventDate in the filter, but again the size of the dataFrame for occurrences is (0, 198):
Query for that case:
There is no last download date given, initial_download should be true, correct? (y/n): >? y
initial_download:  True
Number of occurrences for the Redlands Coast Koala Watch project:    totalRecords
0          2696
Do you want to download all data? (y/n): >? y
payload for queryID: {'fq': ['taxa:"Phascolarctos cinereus"', 'dataResourceUid:"dr19064"']}
queryID URL: https://biocache-ws.ala.org.au/ws/webportal/params
method: POST
qid for query:<PHONE_NUMBER>146
URL for result:https://biocache-ws.ala.org.au/ws/occurrences/offline/download?email=jojo.schultz%40outlook.de&dwcHeaders=True&reasonTypeId=4&emailNotify=false&fq=(qid%3A1709774967146)&fields=images%2Craw_recordedBy%2Cdcterms:modified%2Cdcterms:language%2Cdcterms:license%2CrightsHolder%2Cdcterms:accessRights%2Cdcterms:bibliographicCitation%2Creferences%2CinstitutionID%2CcollectionID%2CdatasetID%2CinstitutionCode%2CcollectionCode%2CdatasetName%2CownerInstitutionCode%2CbasisOfRecord%2CinformationWithheld%2CdataGeneralizations%2CdynamicProperties%2CoccurrenceID%2CcatalogNumber%2CrecordNumber%2CrecordedBy%2CindividualCount%2CorganismQuantity%2CorganismQuantityType%2Csex%2ClifeStage%2CreproductiveCondition%2Cbehavior%2CestablishmentMeans%2CoccurrenceStatus%2Cpreparations%2Cdisposition%2CassociatedMedia%2CassociatedReferences%2CassociatedSequences%2CassociatedTaxa%2CotherCatalogNumbers%2CoccurrenceRemarks%2CorganismID%2CorganismName%2CorganismScope%2CassociatedOccurrences%2CassociatedOrganisms%2CpreviousIdentifications%2CorganismRemarks%2CmaterialSampleID%2CeventID%2CparentEventID%2CfieldNumber%2CeventDate%2CeventTime%2CstartDayOfYear%2CendDayOfYear%2Cyear%2Cmonth%2Cday%2CverbatimEventDate%2Chabitat%2CsamplingProtocol%2CsamplingEffort%2CsampleSizeValue%2CsampleSizeUnit%2CfieldNotes%2CeventRemarks%2ClocationID%2ChigherGeographyID%2ChigherGeography%2Ccontinent%2CwaterBody%2CislandGroup%2Cisland%2Ccountry%2CcountryCode%2CstateProvince%2Ccounty%2Cmunicipality%2Clocality%2CverbatimLocality%2CminimumElevationInMeters%2CmaximumElevationInMeters%2CverbatimElevation%2CminimumDepthInMeters%2CmaximumDepthInMeters%2CverbatimDepth%2CminimumDistanceAboveSurfaceInMeters%2CmaximumDistanceAboveSurfaceInMeters%2ClocationAccordingTo%2ClocationRemarks%2CdecimalLatitude%2CdecimalLongitude%2CgeodeticDatum%2CcoordinateUncertaintyInMeters%2CcoordinatePrecision%2CpointRadiusSpatialFit%2CverbatimCoordinates%2CverbatimLatitude%2CverbatimLongitude%2CverbatimCoordinateSystem%2CverbatimSRS%2CfootprintWKT%2CfootprintSRS%2CfootprintSpatialFit%2CgeoreferencedBy%2CgeoreferencedDate%2CgeoreferenceProtocol%2CgeoreferenceSources%2CgeoreferenceVerificationStatus%2CgeoreferenceRemarks%2CgeologicalContextID%2CearliestEonOrLowestEonothem%2ClatestEonOrHighestEonothem%2CearliestEraOrLowestErathem%2ClatestEraOrHighestErathem%2CearliestPeriodOrLowestSystem%2ClatestPeriodOrHighestSystem%2CearliestEpochOrLowestSeries%2ClatestEpochOrHighestSeries%2CearliestAgeOrLowestStage%2ClatestAgeOrHighestStage%2ClowestBiostratigraphicZone%2ChighestBiostratigraphicZone%2ClithostratigraphicTerms%2Cgroup%2Cformation%2Cmember%2Cbed%2CidentificationID%2CidentificationQualifier%2CtypeStatus%2CidentifiedBy%2CdateIdentified%2CidentificationReferences%2CidentificationVerificationStatus%2CidentificationRemarks%2CtaxonID%2CscientificNameID%2CacceptedNameUsageID%2CparentNameUsageID%2CoriginalNameUsageID%2CnameAccordingToID%2CnamePublishedInID%2CtaxonConceptID%2Craw_scientificName%2CscientificName%2CacceptedNameUsage%2CparentNameUsage%2CoriginalNameUsage%2CnameAccordingTo%2CnamePublishedIn%2CnamePublishedInYear%2ChigherClassification%2Ckingdom%2Cphylum%2Cclass%2Corder%2Cfamily%2Cgenus%2Csubgenus%2CspecificEpithet%2CinfraspecificEpithet%2CtaxonRank%2CverbatimTaxonRank%2CscientificNameAuthorship%2CvernacularName%2CnomenclaturalCode%2CtaxonomicStatus%2CnomenclaturalStatus%2CtaxonRemarks%2CmeasurementDeterminedDate%2Cvideos%2Cspecies%2CrelationshipAccordingTo%2CresourceRelationshipID%2CmeasurementRemarks%2CmeasurementValue%2CrelationshipRemarks%2CresourceID%2CrelationshipEstablishedDate%2CidentifierRole%2CrelationshipOfResource%2CmeasurementMethod%2Cidentifier%2Cprovenance%2Csounds%2CrecordID%2Crights%2Csource%2CmeasurementID%2CmeasurementType%2CmeasurementUnit%2CmeasurementDeterminedBy%2CmeasurementAccuracy%2CrelatedResourceID%2Cimages%2Cdcterms:type%2C&qa=none&flimit=-1
method: GET
Data for download:
https://biocache.ala.org.au/biocache-download/7d58aab5-1e0f-384d-ba9a-f632fb210b51/1709774967617/data.zip
size of dataframe:  (0, 198)
payload for queryID: {'fq': ['taxa:"Phascolarctos cinereus"', 'dataResourceUid:"dr19064"']}
queryID URL: https://biocache-ws.ala.org.au/ws/webportal/params
method: POST
qid for query:<PHONE_NUMBER>402
URL for result:https://biocache-ws.ala.org.au/ws/occurrences/offline/download?email=jojo.schultz%40outlook.de&dwcHeaders=True&reasonTypeId=4&emailNotify=false&fq=(qid%3A1709774968402)&fields=decimalLatitude%2CdecimalLongitude%2CeventDate%2CscientificName%2CtaxonConceptID%2CrecordID%2CdataResourceName%2CoccurrenceStatus%2Cmultimedia%2CmultimediaLicence%2Cimages%2Cvideos%2Csounds%2C&qa=none&flimit=-1
method: GET
Data for download:
https://biocache.ala.org.au/biocache-download/7d58aab5-1e0f-384d-ba9a-f632fb210b51/1709774968804/data.zip
size of media dataframe:  (3806, 17)
Yet the media dataframe size (collect=False) is (3806, 17). Saving this to a csv file works and the images are donwloaded as well if I put collect = True (even though the file can't be opened as format seems to unsupported for some reason.)
Do you have any idea why the dataframe remains empty?
Thanks so much for your consistent support!
Cheers,
Jojo

From: acbuyan @.>
Sent: Thursday, March 7, 2024 08:20
To: AtlasOfLivingAustralia/galah-python @.>
Cc: JojoMimic @.>; Author @.>
Subject: Re: [AtlasOfLivingAustralia/galah-python] is API access with JWT tokens working yet? (Issue #217)
Hi Jojo,
Good to hear from you, and great to hear that you continue to use galah-python! It makes my day :)
First things first: there have been a couple of updates to galah-python; have you updated to the latest version? I don't think this will solve your issue, but I have fixed some bugs along the way and added other options too.
As far as your question on JWT tokens, we’re holding off on JWT token integration until we deploy Cognito. That should be soon but we’re not sure. In addition, JWT tokens are only for access to sensitive data, and only then to access that you have personally been approved to receive. If these conditions do not apply, then JWT tokens are not what's causing your problem.
Unfortunately, galah-python doesn’t support BioCollect APIs, so you can only use galah to get occurrences once they have been passed to biocache.
I may have asked you this before, but what is the reason that you are downloading all fields? It is unfortunately never a good idea to download all of the ALA fields, as they include spatial data that cannot be relevant to every application. It also includes Darwin Core fields that are rarely populated and may not be useful to you. If you know what data fields you need, you can then include it as an argument to atlas_occurrences
—
Reply to this email directly, view it on GitHubhttps://github.com/AtlasOfLivingAustralia/galah-python/issues/217#issuecomment-1981925136, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIGDLPVK3KQLE25G3NFWATDYW6JCVAVCNFSM6AAAAABEIF73CKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBRHEZDKMJTGY.
You are receiving this because you authored the thread.

Hi again Jojo,
I get what you mean about the headings.  If you're not overly familiar with the concept of Darwin Core (https://dwc.tdwg.org/) headings, it can feel overwhelming.
Out of curiosity, have you tried the search_all function in galah? It could help you narrow down your fields search?  You can use show_all to show all the terms, and search_all to find terms with keywords either in the term (id) itself or the description.  It looks like this:
                    id                                     description   type link
0        _nest_parent_                                             NaN  field  NaN
1          _nest_path_                                             NaN  field  NaN
2               _root_                                             NaN  field  NaN
3       abcdTypeStatus                   ABCD field in use by herbaria  field  NaN
4    acceptedNameUsage  http://rs.tdwg.org/dwc/terms/acceptedNameUsage  field  NaN
..                 ...                                             ...    ...  ...
755  multimediaLicence                              Media filter field  media
756             images                              Media filter field  media
757             videos                              Media filter field  media
758             sounds                              Media filter field  media
759                qid                Reference to pre-generated query  other

[760 rows x 4 columns]
>>> galah.search_all(fields="latitude")
                    id                                        description   type                                               link
0      decimalLatitude  The decimal latitude associated with this reco...  field  https://github.com/AtlasOfLivingAustralia/ala-...
1     verbatimLatitude      http://rs.tdwg.org/dwc/terms/verbatimLatitude  field                                                NaN
2  raw_decimalLatitude  The decimal latitude as supplied by the data p...  field                                                NaN

As far as getting the data, I don't think it was a problem with what you were doing, I think it was with how galah-python was constructing the URLs.  I've patched it and pushed my changes to the Python Package Index, so all you should need to do is update your installation and it should work.

Hey,
thanks! Yeah I have tried the search_all, but it's still hard to pick all the fields that I will need, as there are quite a few...
I have upgraded galah-python, but the dataframe is still empty... ?
payload for queryID: {'fq': ['taxa:"Phascolarctos cinereus"', 'dataResourceUid:"dr19064"']}
queryID URL: https://biocache-ws.ala.org.au/ws/webportal/params
method: POST
qid for query:<PHONE_NUMBER>562
URL for result:https://biocache-ws.ala.org.au/ws/occurrences/offline/download?email=jojo.schultz%40outlook.de&dwcHeaders=True&reasonTypeId=4&emailNotify=false&fq=(qid%3A1709852490562)&fields=images%2Craw_recordedBy%2Cdcterms:modified%2Cdcterms:language%2Cdcterms:license%2CrightsHolder%2Cdcterms:accessRights%2Cdcterms:bibliographicCitation%2Creferences%2CinstitutionID%2CcollectionID%2CdatasetID%2CinstitutionCode%2CcollectionCode%2CdatasetName%2CownerInstitutionCode%2CbasisOfRecord%2CinformationWithheld%2CdataGeneralizations%2CdynamicProperties%2CoccurrenceID%2CcatalogNumber%2CrecordNumber%2CrecordedBy%2CindividualCount%2CorganismQuantity%2CorganismQuantityType%2Csex%2ClifeStage%2CreproductiveCondition%2Cbehavior%2CestablishmentMeans%2CoccurrenceStatus%2Cpreparations%2Cdisposition%2CassociatedMedia%2CassociatedReferences%2CassociatedSequences%2CassociatedTaxa%2CotherCatalogNumbers%2CoccurrenceRemarks%2CorganismID%2CorganismName%2CorganismScope%2CassociatedOccurrences%2CassociatedOrganisms%2CpreviousIdentifications%2CorganismRemarks%2CmaterialSampleID%2CeventID%2CparentEventID%2CfieldNumber%2CeventDate%2CeventTime%2CstartDayOfYear%2CendDayOfYear%2Cyear%2Cmonth%2Cday%2CverbatimEventDate%2Chabitat%2CsamplingProtocol%2CsamplingEffort%2CsampleSizeValue%2CsampleSizeUnit%2CfieldNotes%2CeventRemarks%2ClocationID%2ChigherGeographyID%2ChigherGeography%2Ccontinent%2CwaterBody%2CislandGroup%2Cisland%2Ccountry%2CcountryCode%2CstateProvince%2Ccounty%2Cmunicipality%2Clocality%2CverbatimLocality%2CminimumElevationInMeters%2CmaximumElevationInMeters%2CverbatimElevation%2CminimumDepthInMeters%2CmaximumDepthInMeters%2CverbatimDepth%2CminimumDistanceAboveSurfaceInMeters%2CmaximumDistanceAboveSurfaceInMeters%2ClocationAccordingTo%2ClocationRemarks%2CdecimalLatitude%2CdecimalLongitude%2CgeodeticDatum%2CcoordinateUncertaintyInMeters%2CcoordinatePrecision%2CpointRadiusSpatialFit%2CverbatimCoordinates%2CverbatimLatitude%2CverbatimLongitude%2CverbatimCoordinateSystem%2CverbatimSRS%2CfootprintWKT%2CfootprintSRS%2CfootprintSpatialFit%2CgeoreferencedBy%2CgeoreferencedDate%2CgeoreferenceProtocol%2CgeoreferenceSources%2CgeoreferenceVerificationStatus%2CgeoreferenceRemarks%2CgeologicalContextID%2CearliestEonOrLowestEonothem%2ClatestEonOrHighestEonothem%2CearliestEraOrLowestErathem%2ClatestEraOrHighestErathem%2CearliestPeriodOrLowestSystem%2ClatestPeriodOrHighestSystem%2CearliestEpochOrLowestSeries%2ClatestEpochOrHighestSeries%2CearliestAgeOrLowestStage%2ClatestAgeOrHighestStage%2ClowestBiostratigraphicZone%2ChighestBiostratigraphicZone%2ClithostratigraphicTerms%2Cgroup%2Cformation%2Cmember%2Cbed%2CidentificationID%2CidentificationQualifier%2CtypeStatus%2CidentifiedBy%2CdateIdentified%2CidentificationReferences%2CidentificationVerificationStatus%2CidentificationRemarks%2CtaxonID%2CscientificNameID%2CacceptedNameUsageID%2CparentNameUsageID%2CoriginalNameUsageID%2CnameAccordingToID%2CnamePublishedInID%2CtaxonConceptID%2Craw_scientificName%2CscientificName%2CacceptedNameUsage%2CparentNameUsage%2CoriginalNameUsage%2CnameAccordingTo%2CnamePublishedIn%2CnamePublishedInYear%2ChigherClassification%2Ckingdom%2Cphylum%2Cclass%2Corder%2Cfamily%2Cgenus%2Csubgenus%2CspecificEpithet%2CinfraspecificEpithet%2CtaxonRank%2CverbatimTaxonRank%2CscientificNameAuthorship%2CvernacularName%2CnomenclaturalCode%2CtaxonomicStatus%2CnomenclaturalStatus%2CtaxonRemarks%2CmeasurementDeterminedDate%2Cvideos%2Cspecies%2CrelationshipAccordingTo%2CresourceRelationshipID%2CmeasurementRemarks%2CmeasurementValue%2CrelationshipRemarks%2CresourceID%2CrelationshipEstablishedDate%2CidentifierRole%2CrelationshipOfResource%2CmeasurementMethod%2Cidentifier%2Cprovenance%2Csounds%2CrecordID%2Crights%2Csource%2CmeasurementID%2CmeasurementType%2CmeasurementUnit%2CmeasurementDeterminedBy%2CmeasurementAccuracy%2CrelatedResourceID%2Cimages%2Cdcterms:type&qa=none&flimit=-1
method: GET
Data for download:
https://biocache.ala.org.au/biocache-download/7d58aab5-1e0f-384d-ba9a-f632fb210b51/1709852491183/data.zip
size of occurrences dataframe:  (0, 198)
Is there still something wrong with the URL?
Thanks for such prompt help 🙂
Jojo

From: acbuyan @.>
Sent: Friday, March 8, 2024 08:57
To: AtlasOfLivingAustralia/galah-python @.>
Cc: JojoMimic @.>; Author @.>
Subject: Re: [AtlasOfLivingAustralia/galah-python] is API access with JWT tokens working yet? (Issue #217)
Hi again Jojo,
I get what you mean about the headings. If you're not overly familiar with the concept of Darwin Core (https://dwc.tdwg.org/) headings, it can feel overwhelming.
Out of curiosity, have you tried the search_all function in galah? It could help you narrow down your fields search? You can use show_all to show all the terms, and search_all to find terms with keywords either in the term (id) itself or the description. It looks like this:
                id                                     description   type link

0        nest_parent                                             NaN  field  NaN
1          nest_path                                             NaN  field  NaN
2               root                                             NaN  field  NaN
3       abcdTypeStatus                   ABCD field in use by herbaria  field  NaN
4    acceptedNameUsage  http://rs.tdwg.org/dwc/terms/acceptedNameUsage  field  NaN
..                 ...                                             ...    ...  ...
755  multimediaLicence                              Media filter field  media
756             images                              Media filter field  media
757             videos                              Media filter field  media
758             sounds                              Media filter field  media
759                qid                Reference to pre-generated query  other
[760 rows x 4 columns]

galah.search_all(fields="latitude")
id                                        description   type                                               link
0      decimalLatitude  The decimal latitude associated with this reco...  field  https://github.com/AtlasOfLivingAustralia/ala-...
1     verbatimLatitude      http://rs.tdwg.org/dwc/terms/verbatimLatitude  field                                                NaN
2  raw_decimalLatitude  The decimal latitude as supplied by the data p...  field                                                NaN

As far as getting the data, I don't think it was a problem with what you were doing, I think it was with how galah-python was constructing the URLs. I've patched it and pushed my changes to the Python Package Index, so all you should need to do is update your installation and it should work.
—
Reply to this email directly, view it on GitHubhttps://github.com/AtlasOfLivingAustralia/galah-python/issues/217#issuecomment-1984723718, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIGDLPRZ65ZNOR33CLAPSV3YXDWD5AVCNFSM6AAAAABEIF73CKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBUG4ZDGNZRHA.
You are receiving this because you authored the thread.Message ID: @.***>

yes - do you mind double-checking what version of galah-python you have?  It should be 0.8.3.  If it is that version and it is still not working, then I'll have to go back to the drawing board.

Yup it's 0.8.3!
Yesterday's was 0.8.2...
Sorry!
I will reduce the number fo fields a bit more too.
Let me know if I can help with debugging!

From: acbuyan @.>
Sent: Friday, March 8, 2024 09:15
To: AtlasOfLivingAustralia/galah-python @.>
Cc: JojoMimic @.>; Author @.>
Subject: Re: [AtlasOfLivingAustralia/galah-python] is API access with JWT tokens working yet? (Issue #217)
yes - do you mind double-checking what version of galah-python you have? It should be 0.8.3. If it is that version and it is still not working, then I'll have to go back to the drawing board.
—
Reply to this email directly, view it on GitHubhttps://github.com/AtlasOfLivingAustralia/galah-python/issues/217#issuecomment-1984762666, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIGDLPTE2PBNWTOW5JKFJ6DYXDYJ5AVCNFSM6AAAAABEIF73CKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBUG43DENRWGY.
You are receiving this because you authored the thread.Message ID: @.***>

Ok, this might sound weird, but ... try putting decimalLatitude at the beginning of your list of terms and see if that works

That works!
Interesting fix :)
What did that change?

I don't know - I'll ask the systems team and see why.

Thank you!

Hi Jojo,
Wanted to let you know that it is on the ALA side of things.  I'm not sure when a fix is coming, so for now keep using the decimalLatitude fix.  I'm going to close the issue for now, but do let me know if anything else comes up!


Update Xen installation guide

Documentation for Xen installation is not correct. Some of the steps says, update file but gives no information about what to update. The gluster volume mount path is incorrect.
cc: @treddington @theryecatcher

Updated the Xen Installation Wiki.


manifest is not a Parquet file. expected magic number

Hello ,
while trying to read delta table using hive i got the following error :
Caused by: java.lang.RuntimeException: hdfs:/.../_symlink_format_manifest/manifest is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [117, 101, 116, 10].
while searching for the cause of the issue is fixed in spark 3 and delta 0.7.0 see #365 .
i'm using the following commands to create external table in hive :
DeltaTable.forPath(spark, deltaLakeSilverPath).generate("symlink_format_manifest")
CREATE EXTERNAL TABLE IF NOT EXISTS $table_name ($schema) " +       "ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' " +       "STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' " +       "OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' " +       "LOCATION "+symlink_format_manifest_Path
I'm using spark version 3.1.2 and delta 1.0.0 .

Hive does not actually support reading non-text tables using "org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat". This feature was first designed in Hive to work text formatted tables, but was never extended to work with non-text formatted tables. But other systems like Presto extended it to work with Parquet, etc.

@tdas  Thx a lot for the clarification, i just deploy presto and tested it work like charm .


Not on packagist?

Hi,
I was interested to try this package out, but it seems you haven't published to Packagist yet?
Hope to try it soon.
Thanks 👍

Hi @JayBizzle
Sorry for the delay!
The package is now available on Packagist: composer require zbiller/laravel-revisions.
Thanks! :)

👍


Release num-complex 0.2 to crates.io.

I was wondering what development is still necessary on num-complex before a release to crates.io can happen? I'd love to see the recent changes released as this would allow me to make nalgebra support #![no_std] as well!

Mostly, I'm just hesitant to call it ready. 😅 But if it's working for nalgebra, that's a pretty good vote of confidence.  I'll go over it again today and see if there's anything else I think needs a breaking change.

Published!

Great, thanks !


Need to implement CodeSystem versioning for LOINC.

Need to implement CodeSystem versioning for LOINC.
Presently:

All of the terminology service operations assume the latest version of a given CodeSystem.
When uploading a new CodeSystem, the old version is deleted from the terminology service tables.
The version of the CodeSystem resource is independent of the version of the underlying CodeSystem in the terminology service tables.

Closing. See #2060


Fix AND and OR entitlement evaluation in go to propogate errors back to the caller

Description
It looks like this issue may have contributed to some false negative entitlement evalutions on omega that caused some people to be booted from channels when base was unavailable.

it's fixed


fix(autoware_traffic_light_visualization): fix to visualize correct color and shapes

Description
Related to https://github.com/autowarefoundation/autoware.universe/issues/8389.
The visualization result of after this PR:
Screencast from 2024年08月09日 17時12分07秒.webm
Related links
Parent Issue:

Link

How was this PR tested?
Notes for reviewers
None.
Interface changes
None.

Effects on system behavior
None.

@tzhong518 Could you review this PR as a code owner? :pray:


Method accepting NSError** should have a non-void return value to indicate whether or not an error occurred warnings

LLVM/Clang warnings when compiling due to methods that accept NSError** not returning a BOOL value to indicate if there was an error.
File with error: GONMarkupParser.m
Methods with error:

(void)handleClosingTag:(NSString *)tag error:(NSError **)error (line 426)
(void)handleOpeningTag:(NSString *)tag error:(NSError **)error (line 504)
(void)generateError:(NSError **)error tag:(NSString *)tag (line 567)

Don't mind creating a PR if requested.

That's a good point. I should totally update these methods signature.
Thanks.

Sounds good, thank you for your quick reply.

Thanks to you, I was able to fix a bug ;)


Update tracked parameters late in training

Hi there! I was just wondering if there was any way to update a model whose parameters have been unfrozen after starting training (i.e. at iteration N, I set requires_grad = True and give it a .grad value). If I do this, the unlocked parameters are not displayed in the visualization.
Thanks!

Hey @pedrocolon93 could it be possible for us to look at the code and the dashboard screeenshots? It would be a whole lot easier for people to help you if they could see where the probelm was. Thanks!

Closing for now, we can reopen with further details


Search Plugin - issue when deleting

Describe the Bug
Since version 3.4 or 3.5, when I delete an element from a collection used for the search plugin, the items are not deleted in the search page and reindexing doesn't work either.
(but #9612 worked great in 3.3)
Deleted elements

Elements are still in the search table

Error when you click on the content:

Link to the code that reproduces this issue
https://github.com/ogaraud/payload-search-plugin-bug/tree/search-plugin-bug/test/_community
Reproduction Steps
(I use postgres db)

Start the admin panel  pnpm dev _community
Delete the post content and/or the contact content
See on the 'Search Results' page that none of the elements have been deleted

Which area(s) are affected? (Select all that apply)
plugin: search
Environment Info
Binaries:
  Node: 22.11.0
  npm: N/A
  Yarn: N/A
  pnpm: 9.14.2
Relevant Packages:
  payload: 3.5.0
  next: 15.0.4
  @payloadcms/db-postgres: 3.5.0
  @payloadcms/email-nodemailer: 3.5.0
  @payloadcms/graphql: 3.5.0
  @payloadcms/next/utilities: 3.5.0
  @payloadcms/payload-cloud: 3.5.0
  @payloadcms/plugin-nested-docs: 3.5.0
  @payloadcms/plugin-search: 3.5.0
  @payloadcms/richtext-lexical: 3.5.0
  @payloadcms/translations: 3.5.0
  @payloadcms/ui/shared: 3.5.0
  react: 19.0.0
  react-dom: 19.0.0

Hey @ogaraud,
This is a duplicate of #9443 which I have a PR for. It's an issue limited to db-postgres it seems.


Artificial Neural Network(ANN) model development using Matlab toolbox and calculations using MS excel sheet

We ask that you please be patient as our team works through approving and publishing all articles/tutorials in a timely manner. Allow 1-3 days for a topic to be reviewed and/or approved - allow 3-7 days for an articles to be reviewed and/or published.
Be sure to visit our Resources Page for tools, resources, and example articles to go over.
NOTE: (Please ensure that you have only one open issue + linked pull request at a time. This will ensure that we complete the article in a timely manner from inception to publishing.)
NOTE: We tend to stray away or tend not to publish reviews/comparisons of commercial product offerings.
We're looking for the first 2-3 paragraphs of the article that appropriately summarize what your article will be about.
Artificial Neural Network(ANN) model development using Matlab toolbox and calculations using MS excel sheet
Introduction:
Artificial Neural Network(ANN) is a deep learning model that draws inspiration from the neural structure of the human brain. ANNs are complex structures containing interconnected adaptive elements known as artificial neurons that can perform large computations for knowledge representation. A model is the creation of a virtual representation of real-world systems.
A spreadsheet is a file that exists of cells in rows and columns and can help arrange, calculate and sort data. Data in a spreadsheet can be numeric values, as well as text, formulas, references and functions. Both a spreadsheet and script can be used to analyze and visualize a given data. We can bring data from the spreadsheet to Matlab and the vice versa
In this tutorial, we are going to see the demonstration of how the ANN model is presented for input-output mapping using the ANN toolbox. The generation of the Matlab function and its utilization for the input is also demonstrated. After all these, the final weights stored in the Matlab function are also used for model development in the excel sheet for easy standalone implementation.
Key takeaways:

Presentation of the ANN model for input-output mapping using the ANN toolbox.
Generation of the Matlab function and its utilization for the input.
Using Matlab function for model development in the excel sheet.

References:
Please list links to any published content/research that you intend to use to support/guide this article.
Templates to use as guides

How To Guide Template
Software Review Template
Tutorial Template

Good afternoon and thank you for submitting your article for publishing.
Your article has been entered into our queue and should be reviewed (and published pending any changes) as soon as a peer reviewer is finished with the ones in the queue before it. 🚀
Thank you.

Can you make this a unique project? Not just reading and writing from an Excel sheet.

@hectorkambow I think you can check on this... I have made a unique project.
cc @ahmadmardeni1

@hectorkambow here we have a set of data and we developed a neural network to predict the suitable output for the input data. This prediction is compared to that of the excel datasheet.

@ahmadmardeni1 would you check this, please??