How to Analyze Mobile App Traffic and Reverse Engineer Its Non-Public API

| By

Have you ever wanted to analyze the traffic between a mobile app and its servers or reverse engineer a mobile app’s non-public API? Here’s one way.

The basic principle is to proxy the traffic from the app through a computer you control on which you can capture and analyze traffic. If the app you’re interested in is using an unencrypted protocol like HTTP, this is pretty easy. Just run a proxy on your computer and configure your mobile device to proxy network traffic through your computer’s IP.

Most apps these days, however, use encrypted protocols like HTTPS (or are even required to by default by mobile OSes). Data at the TCP layer and below like IP addresses and port numbers are visible in plaintext, but all application level data at the HTTPS layer is encrypted. So you run a proxy that supports HTTPS on your computer, but then your app doesn’t trust the self-signed TLS certificate your computer presents. Mobile apps used to trust certificates that the mobile device’s system trusted. So you could just download the self-signed certificate onto the mobile device and configure the mobile OS to trust it. But these days mobile app frameworks let developers customize their app’s network security settings (like so for Android).

Let’s say your mobile app has custom trust anchors or pins certificates. What do you do now? You can either

  1. disable the certificate check completely
  2. or alter the certificate check

I’m not familiar with how to do this on iOS (there seem to be good resources out there like this) so will show how to do option two on Android.

Setup mobile device and app

I don’t have an Android so used an emulator called Genymotion. I created a Samsung Galaxy S9 virtual device which is has a recent enough Android OS to run most mobile apps. In order to install the mobile app from the Google Play Store I had to install OpenGApps. I think I’m also able to download the APK from the web and drag and drop it into the emulator to install.

Install Charles Proxy TLS certificate on device

To install the Charles cert, I had to open this page in Chrome. The built-in browser in the emulator didn’t seem to prompt me to download the Charles cert, but Chrome did. I installed Chrome by install OpenGApps and then installing Chrome from the Play store. I think I also needed to configure the Android device to use Charles as its proxy with these steps in order to get the certificate download prompt. Then I made the Android device trust it.

Patch the Android app’s network security config

I used `apktool to decompile the APK.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

brew install apktool

apktool d /path/to/app.apk
cd app
find . -name network_security_config.xml
./res/xml/network_security_config.xml

cat res/xml/network_security_config.xml

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <domain-config cleartextTrafficPermitted="true">
        <domain includeSubdomains="true">res.cloudinary.com</domain>
        <domain includeSubdomains="true">app-res.cloudinary.com</domain>
    </domain-config>
</network-security-config>

The app only allows cleartext to the above two domains. I don’t see any pinned certificates, but there must be some defaults since the app didn’t trust the same certs trusted by the Android OS. So I updated network_security_config.xml to be the following.

1
2
3
4
5
6
7
8
9
10
11
12
13

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <domain-config cleartextTrafficPermitted="true">
        <domain includeSubdomains="true">res.cloudinary.com</domain>
        <domain includeSubdomains="true">app-res.cloudinary.com</domain>
    </domain-config>
    <base-config>
        <trust-anchors>
            <certificates src="system" />
            <certificates src="user" />
        </trust-anchors>
    </base-config>
</network-security-config>

Then I tried recompiling the patched APK but got the following error.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

cd app
apktool b . -o ~/Downloads/app-patched.apk

I: Using Apktool 2.4.1
I: Checking whether sources has changed...
I: Smaling smali folder into classes.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes10 folder into classes10.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes9 folder into classes9.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes7 folder into classes7.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes6 folder into classes6.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes8 folder into classes8.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes3 folder into classes3.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes4 folder into classes4.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes5 folder into classes5.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes2 folder into classes2.dex...
I: Checking whether resources has changed...
I: Building resources...
W: invalid resource directory name: /Users/dxia/Downloads/app/./res navigation
brut.androlib.AndrolibException: brut.common.BrutException: could not exec (exit code = 1): [/var/folders/y_/sjt8100n43g69mtr9t588d6r0000gp/T/brut_util_Jar_15064276297777137207.tmp, p, --forced-package-id, 127, --min-sdk-version, 21, --target-sdk-version, 29, --version-code, 160072564, --version-name, 7.21.0, --no-version-vectors, -F, /var/folders/y_/sjt8100n43g69mtr9t588d6r0000gp/T/APKTOOL339327577576851750.tmp, -e, /var/folders/y_/sjt8100n43g69mtr9t588d6r0000gp/T/APKTOOL5191817693537904820.tmp, -0, arsc, -I, /Users/dxia/Library/apktool/framework/1.apk, -S, /Users/dxia/Downloads/app/./res, -M, /Users/dxia/Downloads/app/./AndroidManifest.xml]

This Github issue comment suggested I run that command with the --use-aapt2 switch. Then I got another error.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

apktool b --use-aapt2 . -o ~/Downloads/app-patched.apk

I: Using Apktool 2.4.1
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether resources has changed...
I: Building resources...
W: /Users/dxia/Downloads/app-patched/./res/values/public.xml:2119: error: resource 'drawable/$avd_hide_password__0' has invalid entry name '$avd_hide_password__0'. Invalid character '$avd_hide_password__0'.
W: /Users/dxia/Downloads/app-patched/./res/values/public.xml:2120: error: resource 'drawable/$avd_hide_password__1' has invalid entry name '$avd_hide_password__1'. Invalid character '$avd_hide_password__1'.
W: /Users/dxia/Downloads/app-patched/./res/values/public.xml:2121: error: resource 'drawable/$avd_hide_password__2' has invalid entry name '$avd_hide_password__2'. Invalid character '$avd_hide_password__2'.
W: /Users/dxia/Downloads/app-patched/./res/values/public.xml:2122: error: resource 'drawable/$avd_show_password__0' has invalid entry name '$avd_show_password__0'. Invalid character '$avd_show_password__0'.
W: /Users/dxia/Downloads/app-patched/./res/values/public.xml:2123: error: resource 'drawable/$avd_show_password__1' has invalid entry name '$avd_show_password__1'. Invalid character '$avd_show_password__1'.
W: /Users/dxia/Downloads/app-patched/./res/values/public.xml:2124: error: resource 'drawable/$avd_show_password__2' has invalid entry name '$avd_show_password__2'. Invalid character '$avd_show_password__2'.
W: error: resource android:style/Animation.InputMethodFancy is private.
W: error: resource android:style/Animation.VoiceInteractionSession is private.
W: error: resource android:style/AlertDialog is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v24/styles.xml:10: error: style attribute 'android:attr/preferenceListStyle' is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v24/styles.xml:40: error: style attribute 'android:attr/preferenceListStyle' is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v24/styles.xml:70: error: style attribute 'android:attr/preferenceListStyle' is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v24/styles.xml:99: error: style attribute 'android:attr/preferenceListStyle' is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v28/styles.xml:8: error: style attribute 'android:attr/allowMassStorage' is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v26/styles.xml:13: error: resource android:attr/internalMaxWidth is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v26/styles.xml:16: error: resource android:attr/internalMaxWidth is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v26/styles.xml:20: error: style attribute 'android:attr/internalMinHeight' is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v28/styles.xml:17: error: resource android:attr/allowMassStorage is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v28/styles.xml:20: error: resource android:attr/allowMassStorage is private.
W: error: resource android:style/DialogWindowTitle is private.
W: /Users/dxia/Downloads/app-patched/./res/values-v23/styles.xml:13: error: style attribute 'android:attr/attr/private_resource_pad36' not found.
W: /Users/dxia/Downloads/app-patched/./res/values-v23/styles.xml:14: error: style attribute 'android:attr/attr/private_resource_pad35' not found.
W: /Users/dxia/Downloads/app-patched/./res/values-v23/styles.xml:20: error: style attribute 'android:attr/attr/private_resource_pad36' not found.
W: /Users/dxia/Downloads/app-patched/./res/values-v23/styles.xml:21: error: style attribute 'android:attr/attr/private_resource_pad35' not found.
W: /Users/dxia/Downloads/app-patched/./res/values-v23/styles.xml:24: error: resource android:attr/private_resource_pad31 not found.
W: /Users/dxia/Downloads/app-patched/./res/values-v26/styles.xml:10: error: style attribute 'android:attr/internalMinHeight' is private.
brut.androlib.AndrolibException: brut.common.BrutException: could not exec (exit code = 1): [/var/folders/y_/sjt8100n43g69mtr9t588d6r0000gp/T/brut_util_Jar_11817644492691338390.tmp, link, -o, /var/folders/y_/sjt8100n43g69mtr9t588d6r0000gp/T/APKTOOL6551307854758959712.tmp, --package-id, 127, --min-sdk-version, 21, --target-sdk-version, 29, --version-code, 160072564, --version-name, 7.21.0, --no-auto-version, --no-version-vectors, --no-version-transitions, --no-resource-deduping, -e, /var/folders/y_/sjt8100n43g69mtr9t588d6r0000gp/T/APKTOOL6723837428467013762.tmp, -0, arsc, -I, /Users/dxia/Library/apktool/framework/1.apk, --manifest, /Users/dxia/Downloads/app-patched/./AndroidManifest.xml, /Users/dxia/Downloads/app-patched/./build/resources.zip]

This PR fixes the above on Linux and Windows. As of this writing, it’s not released yet. So I had to build from source on an Ubuntu VM.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

java -jar ~/Apktool/brut.apktool/apktool-cli/build/libs/apktool-cli-all.jar b --use-aapt2 . -o ~/Downloads/app-patched.apk

I: Using Apktool 2.4.2-3ac7e8-SNAPSHOT
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether sources has changed...
I: Checking whether resources has changed...
I: Building apk file...
I: Copying unknown files/dir...
I: Built apk...

I signed the patched APK. First I generated some keys. I’m not sure if certain signing and key algorithms are required, but these are the ones I used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

keytool -genkey -alias keys -keystore keys -sigalg MD5withRSA -keyalg RSA -keysize 2048 -validity 10000

Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:
What is the name of your organizational unit?
  [Unknown]:
What is the name of your organization?
  [Unknown]:

What is the name of your City or Locality?
  [Unknown]:  What is the name of your State or Province?
  [Unknown]:
What is the two-letter country code for this unit?
  [Unknown]:
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
  [no]:  yes


Warning:
The generated certificate uses the MD5withRSA signature algorithm which is considered a security risk.

jarsigner -sigalg MD5withRSA -digestalg SHA1 -verbose -keystore keys app-patched.apk keys

Then when dragging and dropping the patched APK into the virtual device, I got an error saying the app couldn’t be installed. In these cases, generating the logs and grepping through them for errors like INSTALL_PARSE_FAILED_NO_CERTIFICATES and INSTALL_FAILED_VERIFICATION_FAILURE helps. I fixed this last error by disabling USB verification in the virtual device settings. The setting for this is inside the virtual Android device itself under “developer settings.”

Sniff the traffic

I made sure the traffic was proxied through my computer, the patched app started successfully, and I was able to see unencrypted data in Charles!

References

Comments